{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"01_PT_Linear_Regression","provenance":[{"file_id":"https://github.com/madewithml/lessons/blob/master/notebooks/02_Basics/04_Linear_Regression.ipynb","timestamp":1582568936460}],"collapsed_sections":[],"toc_visible":true},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"7nDdR3N2R2PJ","colab_type":"text"},"source":["# Linear Regression\n","\n","In this lesson we will learn about linear regression. We will understand the basic math behind it, implement it in just NumPy and then in [PyTorch](https://pytorch.org/)."]},{"cell_type":"markdown","metadata":{"id":"Izx6GIn_SETR","colab_type":"text"},"source":["<div align=\"left\">\n","<a href=\"https://github.com/madewithml/lessons/blob/master/notebooks/02_Basics/01_Linear_Regression/01_PT_Linear_Regression.ipynb\" role=\"button\"><img class=\"notebook-badge-image\" src=\"https://img.shields.io/static/v1?label=&amp;message=View%20On%20GitHub&amp;color=586069&amp;logo=github&amp;labelColor=2f363d\"></a>&nbsp;\n","<a href=\"https://colab.research.google.com/github/madewithml/lessons/blob/master/notebooks/02_Basics/01_Linear_Regression/01_PT_Linear_Regression.ipynb\"><img class=\"notebook-badge-image\" src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>\n","</div>"]},{"cell_type":"markdown","metadata":{"id":"NGE2LxIQuSRa","colab_type":"text"},"source":["# Overview"]},{"cell_type":"markdown","metadata":{"id":"Hi-KdxD_uSln","colab_type":"text"},"source":["Our goal is to learn a linear model $\\hat{y}$ that models $y$ given $X$. \n","\n","$\\hat{y} = XW + b$\n","* $\\hat{y}$ = predictions | $\\in \\mathbb{R}^{NX1}$ ($N$ is the number of samples)\n","* $X$ = inputs | $\\in \\mathbb{R}^{NXD}$ ($D$ is the number of features)\n","* $W$ = weights | $\\in \\mathbb{R}^{DX1}$ \n","* $b$ = bias | $\\in \\mathbb{R}^{1}$ "]},{"cell_type":"markdown","metadata":{"id":"DxOri3K3uSsB","colab_type":"text"},"source":["* **Objective:**  Use inputs $X$ to predict the output $\\hat{y}$ using a linear model. The model will be a line of best fit that minimizes the distance between the predicted (model's output) and target (ground truth) values. Training data $(X, y)$ is used to train the model and learn the weights $W$ using gradient descent.\n","* **Advantages:**\n","  * Computationally simple.\n","  * Highly interpretable.\n","  * Can account for continuous and categorical features.\n","* **Disadvantages:**\n","  * The model will perform well only when the data is linearly separable (for classification).\n","  * Usually not used for classification and only for regression.\n","* **Miscellaneous:** You can also use linear regression for binary classification tasks where if the predicted continuous value is above a threshold, it belongs to a certain class. But we will cover better techniques for classification in future lessons and will focus on linear regression for continuous regression tasks only.\n"]},{"cell_type":"markdown","metadata":{"id":"G8EkjWiKd68P","colab_type":"text"},"source":["# Generate data"]},{"cell_type":"markdown","metadata":{"id":"2u0wnvwWgjmT","colab_type":"text"},"source":["We're going to create some simple dummy data to apply linear regression on. It's going to create roughly linear data (`y = 3.5X + noise`); the random noise is added to create realistic data that doesn't perfectly align in a line. Our goal is to have the model converge to a similar linear equation (there will be slight variance since we added some noise)."]},{"cell_type":"code","metadata":{"id":"a39zl86Fg1CL","colab_type":"code","colab":{}},"source":["import numpy as np\n","import pandas as pd\n","import matplotlib.pyplot as plt"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"9rUjxlhvhzIR","colab_type":"code","colab":{}},"source":["SEED = 1234\n","NUM_SAMPLES = 50"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"OOT0wOiZX9eJ","colab_type":"code","colab":{}},"source":["# Set seed for reproducibility\n","np.random.seed(SEED)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"vCcF_QkYdozl","colab_type":"code","colab":{}},"source":["# Generate synthetic data\n","def generate_data(num_samples):\n","    \"\"\"Generate dummy data for linear regression.\"\"\"\n","    X = np.array(range(num_samples))\n","    random_noise = np.random.uniform(-10,20,size=num_samples)\n","    y = 3.5*X + random_noise # add some noise\n","    return X, y"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"kzSANn29g36a","colab_type":"code","outputId":"be6e25b0-38b8-450c-c42c-a4b159617f3f","executionInfo":{"status":"ok","timestamp":1583943488823,"user_tz":420,"elapsed":2936,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":102}},"source":["# Generate random (linear) data\n","X, y = generate_data(num_samples=NUM_SAMPLES)\n","data = np.vstack([X, y]).T\n","print (data[:5])"],"execution_count":122,"outputs":[{"output_type":"stream","text":["[[ 0.         -4.25441649]\n"," [ 1.         12.16326313]\n"," [ 2.         10.13183217]\n"," [ 3.         24.06075751]\n"," [ 4.         27.39927424]]\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"4ckWsUsKhCDC","colab_type":"code","outputId":"31b1b08e-54dd-432d-a216-29966afdd816","executionInfo":{"status":"ok","timestamp":1583943488824,"user_tz":420,"elapsed":2906,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":204}},"source":["# Load into a Pandas DataFrame\n","df = pd.DataFrame(data, columns=['X', 'y'])\n","X = df[['X']].values\n","y = df[['y']].values\n","df.head()"],"execution_count":123,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>X</th>\n","      <th>y</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.0</td>\n","      <td>-4.254416</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1.0</td>\n","      <td>12.163263</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>2.0</td>\n","      <td>10.131832</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3.0</td>\n","      <td>24.060758</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>4.0</td>\n","      <td>27.399274</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["     X          y\n","0  0.0  -4.254416\n","1  1.0  12.163263\n","2  2.0  10.131832\n","3  3.0  24.060758\n","4  4.0  27.399274"]},"metadata":{"tags":[]},"execution_count":123}]},{"cell_type":"code","metadata":{"id":"qh3z65QXhCj5","colab_type":"code","outputId":"0acfe9d5-4813-4a35-d1c6-d1bbe314f965","executionInfo":{"status":"ok","timestamp":1583943489798,"user_tz":420,"elapsed":3778,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":281}},"source":["# Scatter plot\n","plt.title(\"Generated data\")\n","plt.scatter(x=df['X'], y=df['y'])\n","plt.show()"],"execution_count":124,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAbGElEQVR4nO3de5RdZXnH8e/PSdAR0OEyxjAQEzSE\ngtSkTlEbtYgXorUSWS6EqsVKG2+spRXRoF2KdlHSUqV2YbGhUrBLEcolspClpoSKtaBOTMpFoAaE\nkjEkwyUFJY0kPP3j7IHN5JyZc/bZ57L3/n3WOmvOeffe57w7DM955708ryICMzMrl2f1ugJmZpY/\nB3czsxJycDczKyEHdzOzEnJwNzMrIQd3M7MScnA36wBJ75X0Hy2cf6+kN3SyTlYtDu7WNZJOkvQj\nSb+WtC15/iFJ6nXdppL075L+tNf1qEdSSHpJr+th/c3B3bpC0unAl4BzgRcCc4APAEuBvbpcl1nd\n/DyzXnBwt46T9Hzg88CHIuKKiHgsajZExLsiYmdy3rMl/a2k/5G0VdJXJA0mx46RtFnS6Umrf4uk\nP0l9RjPXflLSA8A/S9pP0rWSJiQ9kjw/ODn/bOA1wPmSfiXp/KT8cElrJT0s6S5JJ6Y+/wBJ10h6\nVNKPgRfP8G/yHkn3SXpI0qenHDta0k2Stif3eb6kvZJjNyan/VdSt3dOdy9WXQ7u1g2vAp4NfGuG\n81YBhwGLgZcAI8BnUsdfCDw/KT8V+LKk/Vq4dn/gRcAKar/7/5y8ngfsAM4HiIhPAz8ATouIfSLi\nNEl7A2uBbwAvAE4C/kHSEcn7fxn4P2Au8L7kUVdyzQXAe4CDgAOAdDDeDfw5cCC1f7vXAx9K6vba\n5JyXJXW7bLp7sQqLCD/86OgDeDfwwJSy/wS2UwtErwUE/Bp4ceqcVwG/SJ4fk5w7K3V8G/DKJq/9\nDfCcaeq4GHgk9frfgT9NvX4n8IMp1/wj8FlgAHgCODx17K+A/2jwWZ8Bvpl6vXdSvzc0OP+jwNWp\n1wG8pNl78aOaD/c9Wjc8BBwoaVZE7AKIiN8DkLSZWstzGHgusD41vipqgfOp95m8PvE4sE+T105E\nxP89dVB6LnAesAyYbP3vK2kgInbXuYcXAa+QtD1VNgv4l+TzZwH3p47dV/+fAqi11p86NyJ+Lemh\nVN0OA74IjCb3NQtY3+jNMtyLVYC7ZawbbgJ2AsdPc86D1FrmR0bEUPJ4fkTs08T7N3Pt1PSnpwOL\ngFdExPOo/fUAtS+FeuffD3w/9f5DUesW+SAwAewCDkmdP2+a+m5Jn5sE5wNSxy8A7gQWJnX7VKpe\n9cx0L1ZBDu7WcRGxHfgctT7qd0jaV9KzJC2m1iVBRDwJXAicJ+kFAJJGJB3XxPtnuXZfal8I2yXt\nT617JW0rcGjq9bXAYclA6Ozk8buSfitpHV8FnCXpuUmf+inTfPYVwFslvToZKP08z/x/cV/gUeBX\nkg4HPjhD3Wa6F6sgB3frioj4G+BjwCeoBaet1PqsP0mt/53k+SbgZkmPAv9GrUXajFav/TtgkFqr\n/2bgO1OOfwl4RzL75O8j4jHgTdQGUn8JPAD8NbWBYoDTqHURPQBcTG2As66IuB34MLXB2S3AI8Dm\n1CkfB/4IeIzal9ZlU97iLOCSZDbNiU3ci1WQIrxZh5lZ2bjlbmZWQg7uZmYl5OBuZlZCDu5mZiXU\nF4uYDjzwwJg/f36vq2FmVijr169/MCKG6x3ri+A+f/58xsbGel0NM7NCkdRwJbS7ZczMSsjB3cys\nhBzczcxKyMHdzKyEHNzNzEqoL2bLmJlVzZoN45z73bv45fYdHDQ0yBnHLWL5kpHc3t/B3cysy9Zs\nGOfMq25lxxO1vVTGt+/gzKtuBcgtwM/YLSPpomRD4ttSZZdJ2pg87pW0MSmfL2lH6thXcqmlmVmJ\nnPvdu54K7JN2PLGbc797V26f0UzL/WJqm+1+bbIgIt45+VzSF4D/TZ1/d0QszquCZmZl88vtO1oq\nz2LGlntE3Ag8XO+YahtWnghcmluNzMxK7qChwZbKs2h3tsxrgK0R8fNU2QJJGyR9X9JrGl0oaYWk\nMUljExMTbVbDzKw4zjhuEYOzB55RNjh7gDOOa3bjsZm1O6B6Ms9stW8B5kXEQ5JeDqyRdGREPDr1\nwohYDawGGB0d9XZQZlZYrc58mTzWl7NlJM0CTgBePlkWETup7XJPRKyXdDdwGOCsYGZWSllnvixf\nMpJrMJ+qnW6ZNwB3RsRTG/tKGpY0kDw/FFgI3NNeFc3M+lc3Zr5k0cxUyEuBm4BFkjZLOjU5dBJ7\nDqS+FrglmRp5BfCBiKg7GGtmVgbdmPmSxYzdMhFxcoPy99YpuxK4sv1qmZkVw0FDg4zXCeSTM186\nvRK1EeeWMTNrw3QzXyb748e37yB4uj9+zYbxjtfLwd3MrA3Ll4xwzglHMTI0iICRoUHOOeEoli8Z\n6Wl/vHPLmJm1qdHMl172x7vlbmbWId1YidqIg7uZ2RRrNoyzdNU6Fqz8NktXrcvcR96NlaiNuFvG\nzCqr3kwWILd0vN1YidqIInq/8n90dDTGxryI1cy6Z+rKUqi1qp8z+1k88vgTe5w/MjTID1ce280q\nzkjS+ogYrXfMLXczK716LfRGM1mmlk3q9aKkVjm4m1mpNcr90iiIN9KNQdA8eUDVzEqtUQt9QKp7\n/tDg7J4NgubJLXczK7VG3Sm7IxicPbBHn/tZbzsS6M0gaJ4c3M2s1BrlfhlJ9b3XC+JFC+ZTObib\nWamdcdyiurNiJgN50YN4Iw7uZlZqvZxr3ksO7mbWU91IiVvmFnojDu5m1jNZt6izmTm4m1nPTJcS\ntx+De6823sjCwd3MeqZft6irp2h/ZTSzh+pFkrZJui1VdpakcUkbk8dbUsfOlLRJ0l2SjutUxc2s\n+HqZErdV/boRdiPNrFC9GFhWp/y8iFicPK4DkHQEtY2zj0yu+QdJA3WuNTPraUrcVhXprwxoIrhH\nxI3Aw02+3/HANyNiZ0T8AtgEHN1G/cysxKbboq7fFOmvDGivz/00SX8MjAGnR8QjwAhwc+qczUnZ\nHiStAFYAzJs3r41qmFm/yDLgWJRpitMthupHWROHXQC8GFgMbAG+0OobRMTqiBiNiNHh4eGM1TCz\nfjE54Di+fQfB0wOOWXcx6jdF+isDMrbcI2Lr5HNJFwLXJi/HgUNSpx6clJlZyRVtWmMWRfkrAzK2\n3CXNTb18OzA5k+Ya4CRJz5a0AFgI/Li9KppZERRtwLHsZmy5S7oUOAY4UNJm4LPAMZIWAwHcC7wf\nICJul3Q58DNgF/DhiGgtI76ZFVKj7Iv9OuBYdjMG94g4uU7xV6c5/2zg7HYqZWbFU7QBx7LzClUz\ny0VVsy/2Kwd3M8tNkQYcy857qJqZlZBb7mZWKEXKzNhLDu5mVhjTZWYE9/enObibWWE0Wih11jW3\ns3PXk4VJx9sNDu5m1hV5dKc0WhC1fccTe5SVbXVsqzygamYdl1femVYXRFV5dayDu5l1XF4bXTTK\n/77fc2fXPb/Kq2PdLWNmDeU1MyWvvDONFkoBXh07hYO7mdWV556heeadmW6hlGfLPM3B3czqyjOF\nb5a8M63+1eDVsc/k4G5mdeWZwrfVvDN5/tVQVQ7uZla3lZx3Ct9WWtZV2Pij0zxbxqziGk1TfN3h\nw3VnpnRjkNIbf7TPwd2s4hq1km+4c6Jne4Y2+uugylMbW+VuGbOKm66V3KtBSm/80T4Hd7OSaXWW\nST9uj+eNP9rXzB6qFwFvBbZFxEuTsnOBPwR+A9wN/ElEbJc0H7gDmFx2dnNEfKAD9TazOrLMMunX\nVrKnNranmT73i4FlU8rWAi+NiN8G/hs4M3Xs7ohYnDwc2M26KMsy/+VLRnrWt26d08wG2TcmLfJ0\n2fdSL28G3pFvtcwsi6yzTNxKLp88+tzfB1yWer1A0gbgUeAvIuIHOXyGmTWhW/3n3g2p/7U1FVLS\np4FdwNeToi3AvIhYAnwM+Iak5zW4doWkMUljExMT7VTDzBKNsibm2X+eV/pe66zMwV3Se6kNtL4r\nIgIgInZGxEPJ8/XUBlsPq3d9RKyOiNGIGB0eHs5aDbPSW7NhnKWr1rFg5bdZumrdtEG0G/3neaXv\ntc7K1C0jaRnwCeD3I+LxVPkw8HBE7JZ0KLAQuCeXmppVUJbZL53uP/fq0WKYseUu6VLgJmCRpM2S\nTgXOB/YF1kraKOkryemvBW6RtBG4AvhARDzcobqblV4/tpK9erQYmpktc3Kd4q82OPdK4Mp2K2Vm\nNf3YSu7XefH2TM4tY9bH+rGV7HnxxeD0A2Z9rF9byZ4X3/8c3M36mHOsWFYO7mZ9zq1ky8J97mZm\nJeTgbmZWQu6WMSso53ex6Ti4mxVQlpWrVi0O7mYFNN3K1emCu1v71eHgblZAWVauurVfLQ7uZh2Q\npYXcyjVZ8rZnbe1bMXm2jFnOsuQ7b/WaLHnb+zFPjXWOg7tZzrJkcmz1miz5XfoxT411jrtlzHKW\npYWc5ZpWV672a54a6wy33M1ylqWF3I1WtbM5Votb7mY5y9JC7lar2nlqqsPB3SxnWTI5Ovuj5U3J\n3tY9NTo6GmNjY72uhpkX+VihSFofEaP1jrnlbpbwIh8rk6YGVCVdJGmbpNtSZftLWivp58nP/ZJy\nSfp7SZsk3SLpdzpVebM89eNm1GZZNTtb5mJg2ZSylcD1EbEQuD55DfBmYGHyWAFc0H41zTrPi3ys\nTJoK7hFxI/DwlOLjgUuS55cAy1PlX4uam4EhSXPzqKxZJ3mRj5VJO33ucyJiS/L8AWBO8nwEuD91\n3uakbEuqDEkrqLXsmTdvXhvVMMvHdNMR8xpo9YCtdUsuA6oREZJamnYTEauB1VCbLZNHPcza0Wg6\nIpDLQKsHbK2b2gnuWyXNjYgtSbfLtqR8HDgkdd7BSZlZ36u3yGfpqnW5ZFN0VkbrpnbSD1wDnJI8\nPwX4Vqr8j5NZM68E/jfVfWNWOHkNtHrA1rqp2amQlwI3AYskbZZ0KrAKeKOknwNvSF4DXAfcA2wC\nLgQ+lHutzboor4FWD9haNzXVLRMRJzc49Po65wbw4XYqZdZP8hpodVZG6yavUDWbQV4Drc4fY93k\n3DJmGS1dta7uVncjQ4P8cOWxPaiRVc10uWWcz90sIw+QWj9zcDfLyAOk1s8c3M0yyrJJtVm3eEDV\nLCMPkFo/c3A3a4O3rbN+5W4ZM7MScnA3MyshB3czsxJyn7v1Hec8N2ufg7v1lSw5z1v9MvCXh1WB\nu2Wsr7S6SfXkl8H49h0ET38ZrNlQfwuBVs83KyoHd+srrS7pb/XLoNXzzYrKwd36SqtL+lv9MnA+\nGKsKB3frK60u6W/1y8D5YKwqHNytryxfMsI5JxzFyNAgopY+95wTjpp2A4xWvgycD8aqwrNlrCem\nm7HSypL+VvO7OB+MVUXmzTokLQIuSxUdCnwGGAL+DJhIyj8VEddN917erKNapk53hFrreboWupnt\nqSObdUTEXRGxOCIWAy8HHgeuTg6fN3lspsBu1eMZK2adl1ef++uBuyPivpzez0rMM1bMOi+v4H4S\ncGnq9WmSbpF0kaT96l0gaYWkMUljExMT9U6xkvKMFbPOazu4S9oLeBvwr0nRBcCLgcXAFuAL9a6L\niNURMRoRo8PDw+1WwwrEM1bMOi+P2TJvBn4aEVsBJn8CSLoQuDaHz7ASyTpjxTlhzJqXR3A/mVSX\njKS5EbElefl24LYcPsNKptUdjLIkFDOrsraCu6S9gTcC708V/42kxUAA9045ZpbJdDNs8soWaVYm\nbQX3iPg1cMCUsve0VSOzOlqdYeOWvlWd0w9YIbQ6w8Zz6a3qHNytEFqdYeO59FZ1Du5WCK0mFPNc\neqs6Jw6zwmhlhs0Zxy2qm7/Gc+mtKhzcrZSc/dGqzsHdSqvVufRmZeLgbrnwnHKz/uLgbm3znHKz\n/uPgbi2p10LPsnrUzDrLwd2a1qiFPjWwT/KccrPe8Tx3a1qjFvqAVPd8zyk36x233Cus1UHQRi3x\n3REMzh7wnHKzPuKWe0VNdrGMb99B8HQXy5oN4w2vadQSn1wt2uzqUTPrPLfcKyrLIOh0qz49p9ys\nvzi4V1SWxFpe9WlWHA7uFXXQ0CDjdQL5TIOgbqGbFYODewXUGzh1Yi2zcvOAask1GjgFPAhqVmJt\nt9wl3Qs8BuwGdkXEqKT9gcuA+dT2UT0xIh5p97OsddMNnP5w5bHef9SspPLqlnldRDyYer0SuD4i\nVklambz+ZE6fZS3IMnCaZ64Yf0mY9UanumWOBy5Jnl8CLO/Q59gMsuxIlNf+o1nm0ptZPvII7gF8\nT9J6SSuSsjkRsSV5/gAwJ4fPqZQ1G8ZZumodC1Z+m6Wr1mUOiK3uPQr57T/qTarNeiePbplXR8S4\npBcAayXdmT4YESEppl6UfBGsAJg3b14O1Simet0WQG7dIlnmpmedJjmVN6k26522g3tEjCc/t0m6\nGjga2CppbkRskTQX2FbnutXAaoDR0dE9gn8VNOrbfs7sZ+WaQrfVuel5TZPM60vCzFrXVreMpL0l\n7Tv5HHgTcBtwDXBKctopwLfa+ZyyatRt8cjjT9Q9v1st3uVLRnKZJpmlS8jM8tFuy30OcLVqKV9n\nAd+IiO9I+glwuaRTgfuAE9v8nFJqNVh3s8Wbx0pUpysw6522gntE3AO8rE75Q8Dr23nvKmjUbTE0\nOJudu55suVukH6cdOl2BWW94hWoPNeq2OOttR7bcLeJph2aW5twyPTRTt0UrLV7vY2pmaQ7uPZZX\nt4WnHZpZmrtlSiLLSlQzKy8H9y7Ja8VpI552aGZp7pbpgjwTcTXiaYdmlubg3gXdGuz0tEMzm+Tg\n3gV5D3b243x2M+sv7nPvgjwHOz2f3cya4eDeBXkOdjqNrpk1w90yXZDnYKfns5tZMxzcuySvwU6n\n0TWzZrhbpmA8n93MmuGWe8F4PruZNcPBvYA8n93MZuJuGTOzEnLLfRpeLGRmReXg3kA38sGYmXWK\nu2Ua8GIhMyuyzMFd0iGSbpD0M0m3S/pIUn6WpHFJG5PHW/Krbvd4sZCZFVk73TK7gNMj4qeS9gXW\nS1qbHDsvIv62/ep1R72+dS8WMrMiy9xyj4gtEfHT5PljwB1A4TqjGyXiet3hw14sZGaFlUufu6T5\nwBLgR0nRaZJukXSRpP0aXLNC0piksYmJiTyqkUmjvvUb7pzgnBOOYmRoEAEjQ4Occ8JRHkw1s0JQ\nRLT3BtI+wPeBsyPiKklzgAeBAP4SmBsR75vuPUZHR2NsbKytemS1YOW3qfcvIOAXq/6g29UxM2ua\npPURMVrvWFstd0mzgSuBr0fEVQARsTUidkfEk8CFwNHtfEaneWNpMyujdmbLCPgqcEdEfDFVPjd1\n2tuB27JXr/PyTsTV6Y2wzcya0c5smaXAe4BbJW1Myj4FnCxpMbVumXuB97dVww7LMxGXFz6ZWb9o\nu889D73sc8/T0lXr6k6fHBka5Icrj+1BjcyszDrW527P5IVPZtYvnFsmIy98MrN+5pZ7Bl74ZGb9\nzsE9Ay98MrN+526ZDKbrW/cuSWbWD0oZ3Du9yYb71s2s35WuW6ZRf3iei4nyXvhkZpa30rXcZ9pk\nI48WfZ4Ln8zMOqF0i5gaJQKDWus6HfgHZw94wNPMCqtSi5ga9XsPSN42z8wqo3TBvVF/+O4Gf6F4\n9aiZlVHpgvvyJSN155qPOLWvmVVI6QZUgYZzzdMZG8EzXMysvEoZ3OvxDBczq5LKBHdo3KI3Myub\n0vW5m5mZg7uZWSk5uJuZlVDHgrukZZLukrRJ0spOfY6Zme2pI8Fd0gDwZeDNwBHUNs0+ohOfZWZm\ne+pUy/1oYFNE3BMRvwG+CRzfoc8yM7MpOhXcR4D7U683J2VPkbRC0piksYmJiQ5Vw8ysmno2oBoR\nqyNiNCJGh4eHe1UNM7NS6tQipnHgkNTrg5OyXHV6xyUzs6LqVHD/CbBQ0gJqQf0k4I/y/IDJHZcm\nc8VM7rgEOMCbWeV1pFsmInYBpwHfBe4ALo+I2/P8jJl2XDIzq7KO5ZaJiOuA6zr1/o3ysDs/u5lZ\ngVeoNsrD7vzsZmYFDu6NdlxyfnYzswKn/HV+djOzxgob3MH52c3MGilst4yZmTXm4G5mVkIO7mZm\nJeTgbmZWQg7uZmYlpIjodR2QNAHc18ZbHAg8mFN1isT3XS2+72pp5r5fFBF10+r2RXBvl6SxiBjt\ndT26zfddLb7vamn3vt0tY2ZWQg7uZmYlVJbgvrrXFegR33e1+L6rpa37LkWfu5mZPVNZWu5mZpbi\n4G5mVkKFDu6Slkm6S9ImSSt7XZ9OknSRpG2SbkuV7S9praSfJz/362Ud8ybpEEk3SPqZpNslfSQp\nL/t9P0fSjyX9V3Lfn0vKF0j6UfL7fpmkvXpd106QNCBpg6Rrk9dVue97Jd0qaaOksaQs8+96YYO7\npAHgy8CbgSOAkyUd0dtaddTFwLIpZSuB6yNiIXB98rpMdgGnR8QRwCuBDyf/jct+3zuBYyPiZcBi\nYJmkVwJ/DZwXES8BHgFO7WEdO+kj1PZenlSV+wZ4XUQsTs1vz/y7XtjgDhwNbIqIeyLiN8A3geN7\nXKeOiYgbgYenFB8PXJI8vwRY3tVKdVhEbImInybPH6P2P/wI5b/viIhfJS9nJ48AjgWuSMpLd98A\nkg4G/gD4p+S1qMB9TyPz73qRg/sIcH/q9eakrErmRMSW5PkDwJxeVqaTJM0HlgA/ogL3nXRNbAS2\nAWuBu4HtEbErOaWsv+9/B3wCeDJ5fQDVuG+ofYF/T9J6SSuSssy/64XeicmeFhEhqZTzWiXtA1wJ\nfDQiHq015mrKet8RsRtYLGkIuBo4vMdV6jhJbwW2RcR6Scf0uj498OqIGJf0AmCtpDvTB1v9XS9y\ny30cOCT1+uCkrEq2SpoLkPzc1uP65E7SbGqB/esRcVVSXPr7nhQR24EbgFcBQ5ImG2Rl/H1fCrxN\n0r3UulmPBb5E+e8bgIgYT35uo/aFfjRt/K4XObj/BFiYjKTvBZwEXNPjOnXbNcApyfNTgG/1sC65\nS/pbvwrcERFfTB0q+30PJy12JA0Cb6Q23nAD8I7ktNLdd0ScGREHR8R8av8/r4uId1Hy+waQtLek\nfSefA28CbqON3/VCr1CV9BZqfXQDwEURcXaPq9Qxki4FjqGWBnQr8FlgDXA5MI9ayuQTI2LqoGth\nSXo18APgVp7ug/0UtX73Mt/3b1MbPBug1gC7PCI+L+lQai3a/YENwLsjYmfvato5SbfMxyPirVW4\n7+Qer05ezgK+ERFnSzqAjL/rhQ7uZmZWX5G7ZczMrAEHdzOzEnJwNzMrIQd3M7MScnA3MyshB3cz\nsxJycDczK6H/B8KVIz4zdljAAAAAAElFTkSuQmCC\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"tags":[]}}]},{"cell_type":"markdown","metadata":{"id":"Myxvm0YWlppI","colab_type":"text"},"source":["# NumPy"]},{"cell_type":"markdown","metadata":{"id":"aHhypQvalsTd","colab_type":"text"},"source":["Now that we have our data prepared, we'll first implement linear regression using just NumPy. This will let us really understand the underlying operations."]},{"cell_type":"markdown","metadata":{"id":"kzjHPncuQpuE","colab_type":"text"},"source":["## Split data"]},{"cell_type":"markdown","metadata":{"id":"ElbTBdP8Q6k5","colab_type":"text"},"source":["Since our task is a regression task, we will randomly split our dataset into **three** sets: train, validation and test data splits.\n","\n","* train: used to train our model.\n","* val : used to validate our model's performance during training.\n","* test: used to do an evaluation of our fully trained model."]},{"cell_type":"code","metadata":{"id":"rrTd9wP9Qv2_","colab_type":"code","colab":{}},"source":["TRAIN_SIZE = 0.7\n","VAL_SIZE = 0.15\n","TEST_SIZE = 0.15\n","SHUFFLE = True"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"w3cWO_EwRC72","colab_type":"code","colab":{}},"source":["# Shuffle data\n","if SHUFFLE:\n","    indices = list(range(NUM_SAMPLES))\n","    np.random.shuffle(indices)\n","    X = X[indices]\n","    y = y[indices]"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"mhnoMWg_gZme","colab_type":"text"},"source":["**NOTE**: Be careful not to shuffle X and y separately because then the inputs won't correspond to the outputs!"]},{"cell_type":"code","metadata":{"id":"CfmqoOyIRUop","colab_type":"code","colab":{}},"source":["# Split indices\n","train_start = 0\n","train_end = int(0.7*NUM_SAMPLES)\n","val_start = train_end\n","val_end = int((TRAIN_SIZE+VAL_SIZE)*NUM_SAMPLES)\n","test_start = val_end"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"F8JKSPgXSBf6","colab_type":"code","outputId":"02c5bc5f-0ec0-4b51-d5d3-7a0086c2c661","executionInfo":{"status":"ok","timestamp":1583943489800,"user_tz":420,"elapsed":3721,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["# Split data\n","X_train = X[train_start:train_end]\n","y_train = y[train_start:train_end]\n","X_val = X[val_start:val_end]\n","y_val = y[val_start:val_end]\n","X_test = X[test_start:]\n","y_test = y[test_start:]\n","print (f\"X_train: {X_train.shape}, y_train: {y_train.shape}\")\n","print (f\"X_val: {X_val.shape}, y_test: {y_val.shape}\")\n","print (f\"X_test: {X_test.shape}, y_test: {y_test.shape}\")"],"execution_count":128,"outputs":[{"output_type":"stream","text":["X_train: (35, 1), y_train: (35, 1)\n","X_val: (7, 1), y_test: (7, 1)\n","X_test: (8, 1), y_test: (8, 1)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"ok62daFIQv9x","colab_type":"text"},"source":["## Standardize data"]},{"cell_type":"markdown","metadata":{"id":"0pQweIQuUTpl","colab_type":"text"},"source":["We need to standardize our data (zero mean and unit variance) or our models can optimize quickly when we are training.\n","\n","$z = \\frac{x_i - \\mu}{\\sigma}$\n","* $z$ = standardized value\n","* $x_i$ = inputs\n","* $\\mu$ = mean\n","* $\\sigma$ = standard deviation"]},{"cell_type":"code","metadata":{"id":"pM1h8CrbQyTG","colab_type":"code","colab":{}},"source":["def standardize_data(data, mean, std):\n","    return (data - mean)/std"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"HWvKMt2-Ui3G","colab_type":"code","colab":{}},"source":["# Determine means and stds\n","X_mean = np.mean(X_train)\n","X_std = np.std(X_train)\n","y_mean = np.mean(y_train)\n","y_std = np.std(y_train)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"XM1EXQFeU3cL","colab_type":"text"},"source":["We need to treat the validation and test sets as if they were hidden datasets. So we only use the train set to determine the mean and std to avoid biasing our training process."]},{"cell_type":"code","metadata":{"id":"csHkTwpyUvt4","colab_type":"code","colab":{}},"source":["# Standardize\n","X_train = standardize_data(X_train, X_mean, X_std)\n","y_train = standardize_data(y_train, y_mean, y_std)\n","X_val = standardize_data(X_val, X_mean, X_std)\n","y_val = standardize_data(y_val, y_mean, y_std)\n","X_test = standardize_data(X_test, X_mean, X_std)\n","y_test = standardize_data(y_test, y_mean, y_std)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"qKkTM0n1V8SN","colab_type":"code","outputId":"03649f0a-29d5-4e98-b2a6-d1dce6370fb8","executionInfo":{"status":"ok","timestamp":1583943489810,"user_tz":420,"elapsed":3655,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":119}},"source":["# Check (means should be ~0 and std should be ~1)\n","print (f\"X_train: mean: {np.mean(X_train, axis=0)[0]:.1f}, std: {np.std(X_train, axis=0)[0]:.1f}\")\n","print (f\"y_train: mean: {np.mean(y_train, axis=0)[0]:.1f}, std: {np.std(y_train, axis=0)[0]:.1f}\")\n","print (f\"X_val: mean: {np.mean(X_val, axis=0)[0]:.1f}, std: {np.std(X_val, axis=0)[0]:.1f}\")\n","print (f\"y_val: mean: {np.mean(y_val, axis=0)[0]:.1f}, std: {np.std(y_val, axis=0)[0]:.1f}\")\n","print (f\"X_test: mean: {np.mean(X_test, axis=0)[0]:.1f}, std: {np.std(X_test, axis=0)[0]:.1f}\")\n","print (f\"y_test: mean: {np.mean(y_test, axis=0)[0]:.1f}, std: {np.std(y_test, axis=0)[0]:.1f}\")"],"execution_count":132,"outputs":[{"output_type":"stream","text":["X_train: mean: -0.0, std: 1.0\n","y_train: mean: 0.0, std: 1.0\n","X_val: mean: -0.5, std: 0.5\n","y_val: mean: -0.6, std: 0.5\n","X_test: mean: -0.6, std: 0.9\n","y_test: mean: -0.6, std: 0.9\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"jyqSl8jyQzKz","colab_type":"text"},"source":["## Weights"]},{"cell_type":"markdown","metadata":{"id":"3xQZ0Hbln6TR","colab_type":"text"},"source":["Our goal is to learn a linear model $\\hat{y}$ that models $y$ given $X$. \n","\n","$\\hat{y} = XW + b$\n","* $\\hat{y}$ = predictions | $\\in \\mathbb{R}^{NX1}$ ($N$ is the number of samples)\n","* $X$ = inputs | $\\in \\mathbb{R}^{NXD}$ ($D$ is the number of features)\n","* $W$ = weights | $\\in \\mathbb{R}^{DX1}$ \n","* $b$ = bias | $\\in \\mathbb{R}^{1}$ "]},{"cell_type":"markdown","metadata":{"id":"Baph5j7xn9XE","colab_type":"text"},"source":["1. Randomly initialize the model's weights $W$."]},{"cell_type":"code","metadata":{"id":"aAOAboM0oFbV","colab_type":"code","colab":{}},"source":["INPUT_DIM = X_train.shape[1] # X is 1-dimensional\n","OUTPUT_DIM = y_train.shape[1] # y is 1-dimensional"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"chU5n3IgkVci","colab_type":"code","outputId":"c4c4ff1c-addd-41ce-ef4e-dabbfa5b607b","executionInfo":{"status":"ok","timestamp":1583943489812,"user_tz":420,"elapsed":3628,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Initialize random weights\n","W = 0.01 * np.random.randn(INPUT_DIM, OUTPUT_DIM)\n","b = np.zeros((1, 1))\n","print (f\"W: {W.shape}\")\n","print (f\"b: {b.shape}\")"],"execution_count":134,"outputs":[{"output_type":"stream","text":["W: (1, 1)\n","b: (1, 1)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"_QjDXSXfWv7Y","colab_type":"text"},"source":["## Model"]},{"cell_type":"markdown","metadata":{"id":"tOqP0jrBoMMd","colab_type":"text"},"source":["2. Feed inputs $X$ into the model to receive the predictions $\\hat{y}$.\n","  * $\\hat{y} = XW + b$"]},{"cell_type":"code","metadata":{"id":"-TBnfN4mkVa6","colab_type":"code","outputId":"b6ebef5f-6e38-4f6c-ca8e-4890e0fdf2f1","executionInfo":{"status":"ok","timestamp":1583943489812,"user_tz":420,"elapsed":3581,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Forward pass [NX1] · [1X1] = [NX1]\n","y_pred = np.dot(X_train, W) + b\n","print (f\"y_pred: {y_pred.shape}\")"],"execution_count":135,"outputs":[{"output_type":"stream","text":["y_pred: (35, 1)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"hcvVc7UkWyMx","colab_type":"text"},"source":["## Loss"]},{"cell_type":"markdown","metadata":{"id":"8EbP8eWUoRqo","colab_type":"text"},"source":["3. Compare the predictions $\\hat{y}$ with the actual target values $y$ using the objective (cost) function to determine the loss $J$. A common objective function for linear regression is mean squarred error (MSE). This function calculates the difference between the predicted and target values and squares it.\n","\n","  * $J(\\theta) = MSE = \\frac{1}{N} \\sum_{i-1}^{N} (y_i - \\hat{y}_i)^2 $\n","    * ${y}$ = ground truth | $\\in \\mathbb{R}^{NX1}$\n","    * $\\hat{y}$ = predictions | $\\in \\mathbb{R}^{NX1}$"]},{"cell_type":"code","metadata":{"id":"h30Y6v5FoPBL","colab_type":"code","outputId":"c8f37ccd-e0c6-47d5-fc2a-38fd79ffdaa7","executionInfo":{"status":"ok","timestamp":1583943489813,"user_tz":420,"elapsed":3551,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Loss\n","N = len(y_train)\n","loss = (1/N) * np.sum((y_train - y_pred)**2)\n","print (f\"loss: {loss:.2f}\")"],"execution_count":136,"outputs":[{"output_type":"stream","text":["loss: 0.99\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"BbzZqCXLWzuJ","colab_type":"text"},"source":["## Gradients"]},{"cell_type":"markdown","metadata":{"id":"yBdMI-Xxoazo","colab_type":"text"},"source":["4. Calculate the gradient of loss $J(\\theta)$ w.r.t to the model weights.\n","  * $J(\\theta) = \\frac{1}{N} \\sum_i (y_i - \\hat{y}_i)^2  = \\frac{1}{N}\\sum_i (y_i - X_iW)^2 $\n","    * $\\frac{\\partial{J}}{\\partial{W}} = -\\frac{2}{N} \\sum_i (y_i - X_iW) X_i = -\\frac{2}{N} \\sum_i (y_i - \\hat{y}_i) X_i$\n","    * $\\frac{\\partial{J}}{\\partial{W}} = -\\frac{2}{N} \\sum_i (y_i - X_iW)1 = -\\frac{2}{N} \\sum_i (y_i - \\hat{y}_i)1$"]},{"cell_type":"code","metadata":{"id":"U5zrZ2j4oPYX","colab_type":"code","colab":{}},"source":["# Backpropagation\n","dW = -(2/N) * np.sum((y_train - y_pred) * X_train)\n","db = -(2/N) * np.sum((y_train - y_pred) * 1)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"_sIznBOQrFtw","colab_type":"text"},"source":["**NOTE**: The gradient is the derivative, or the rate of change of a function. It's a vector that points in the direction of greatest increase of a function. For example the gradient of our loss function ($J$) with respect to our weights ($W$) will tell us how to change W so we can maximize $J$. However, we want to minimize our loss so we subtract the gradient from $W$. "]},{"cell_type":"markdown","metadata":{"id":"zKyRWE7BXKku","colab_type":"text"},"source":["## Update weights"]},{"cell_type":"markdown","metadata":{"id":"4Hs2ikYNpSSd","colab_type":"text"},"source":["5. Update the weights $W$ using a small learning rate $\\alpha$.\n","  * $W = W - \\alpha\\frac{\\partial{J}}{\\partial{W}}$\n","  * $b = b - \\alpha\\frac{\\partial{J}}{\\partial{b}}$"]},{"cell_type":"code","metadata":{"id":"rKuOCcjep3ND","colab_type":"code","colab":{}},"source":["LEARNING_RATE = 1e-1"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"v9E7zB0IoPd0","colab_type":"code","colab":{}},"source":["# Update weights\n","W += -LEARNING_RATE * dW\n","b += -LEARNING_RATE * db"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"hYijMGqvpigf","colab_type":"text"},"source":["**NOTE**: The learning rate $\\alpha$ is a way to control how much we update the weights by. If we choose a small learning rate, it may take a long time for our model to train. However, if we choose a large learning rate, we may overshoot and our training will never converge. The specific learning rate depends on our data and the type of models we use but it's typically good to explore in the range of $[1e^{-8}, 1e^{-1}]$. We'll explore learning rate update stratagies in later lessons."]},{"cell_type":"markdown","metadata":{"id":"rRiMWUARXBM4","colab_type":"text"},"source":["## Training"]},{"cell_type":"markdown","metadata":{"id":"745G2w_rrBZl","colab_type":"text"},"source":["6. Repeat steps 2 - 5 to minimize the loss and train the model."]},{"cell_type":"code","metadata":{"id":"-QpmeHBhrIQI","colab_type":"code","colab":{}},"source":["NUM_EPOCHS = 100"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"89vkuOZdoPcK","colab_type":"code","outputId":"c517a2ce-3d24-4f33-a18c-463ae38accd5","executionInfo":{"status":"ok","timestamp":1583943490085,"user_tz":420,"elapsed":3788,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":187}},"source":["# Initialize random weights\n","W = 0.01 * np.random.randn(INPUT_DIM, OUTPUT_DIM)\n","b = np.zeros((1, ))\n","\n","# Training loop\n","for epoch_num in range(NUM_EPOCHS):\n","\n","    # Forward pass [NX1] · [1X1] = [NX1]\n","    y_pred = np.dot(X_train, W) + b\n","\n","    # Loss\n","    loss = (1/len(y_train)) * np.sum((y_train - y_pred)**2)\n","\n","    # show progress\n","    if epoch_num%10 == 0:\n","        print (f\"Epoch: {epoch_num}, loss: {loss:.3f}\")\n","\n","    # Backpropagation\n","    dW = -(2/N) * np.sum((y_train - y_pred) * X_train)\n","    db = -(2/N) * np.sum((y_train - y_pred) * 1)\n","\n","    # Update weights\n","    W += -LEARNING_RATE * dW\n","    b += -LEARNING_RATE * db"],"execution_count":141,"outputs":[{"output_type":"stream","text":["Epoch: 0, loss: 0.990\n","Epoch: 10, loss: 0.039\n","Epoch: 20, loss: 0.028\n","Epoch: 30, loss: 0.028\n","Epoch: 40, loss: 0.028\n","Epoch: 50, loss: 0.028\n","Epoch: 60, loss: 0.028\n","Epoch: 70, loss: 0.028\n","Epoch: 80, loss: 0.028\n","Epoch: 90, loss: 0.028\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"emgsY5TJXZ7R","colab_type":"text"},"source":["## Evaluation"]},{"cell_type":"code","metadata":{"id":"cNV_o2_arQNU","colab_type":"code","colab":{}},"source":["# Predictions \n","pred_train = W*X_train + b\n","pred_test = W*X_test + b"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"KMj2xv1xrXb5","colab_type":"code","outputId":"916a1283-a85c-4677-e9ee-22504ab9536f","executionInfo":{"status":"ok","timestamp":1583943490086,"user_tz":420,"elapsed":3762,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Train and test MSE\n","train_mse = np.mean((y_train - pred_train) ** 2)\n","test_mse = np.mean((y_test - pred_test) ** 2)\n","print (f\"train_MSE: {train_mse:.2f}, test_MSE: {test_mse:.2f}\")"],"execution_count":143,"outputs":[{"output_type":"stream","text":["train_MSE: 0.03, test_MSE: 0.01\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"59OcpgFArYw_","colab_type":"code","outputId":"00ac0bcf-f58e-49e0-d1d6-18caa3ba782c","executionInfo":{"status":"ok","timestamp":1583943490087,"user_tz":420,"elapsed":3712,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":336}},"source":["# Figure size\n","plt.figure(figsize=(15,5))\n","\n","# Plot train data\n","plt.subplot(1, 2, 1)\n","plt.title(\"Train\")\n","plt.scatter(X_train, y_train, label='y_train')\n","plt.plot(X_train, pred_train, color='red', linewidth=1, linestyle='-', label='model')\n","plt.legend(loc='lower right')\n","\n","# Plot test data\n","plt.subplot(1, 2, 2)\n","plt.title(\"Test\")\n","plt.scatter(X_test, y_test, label='y_test')\n","plt.plot(X_test, pred_test, color='red', linewidth=1, linestyle='-', label='model')\n","plt.legend(loc='lower right')\n","\n","# Show plots\n","plt.show()"],"execution_count":144,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAA3IAAAE/CAYAAAADjvF6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzde3xU9bX///ciRojXIBc1QYRWizcQ\nNHhDrfUWrVYi1lZrK1Yp+uux53jacoR6irbHCooea1vbypd6sK1WWoqRihUvVLFYFSQWVETRgjIo\nIoqKBgzJ+v0xk8xMMpNMMpc9l9fz8ciD/dmzZ/aaBFhZ87mZuwsAAAAAUDh6BR0AAAAAAKB7KOQA\nAAAAoMBQyAEAAABAgaGQAwAAAIACQyEHAAAAAAWGQg4AAAAACgyFHJDnzKzMzLaa2eCgYwEAAEB+\noJADMixSdLV+tZhZY0z7ou6+nrs3u/tu7v5GNuIFACDXMp0rY173aTP7eiZjBfLVTkEHABQbd9+t\n9djM1kqa4O6PJrvezHZy9x25iA0AgHzQ3VwJoCN65IAcM7PrzWyOmf3BzD6S9HUzOzbyKeIWM3vL\nzH5mZuWR63cyMzezIZH27yOP/9XMPjKzf5jZ0ADfEgAAGRWZVvBDM3vdzN41s7vNrDLy2K5mdq+Z\nvRfJm8+YWV8zu0XSaEmzIj17twT7LoDsopADgnGupHsk7SlpjqQdkv5DUn9JYySdIenyTp7/NUk/\nlLSXpDck/U82gwUAIMe+L+l0ScdLGiSpSdKtkccmKDyqrFrhvHmlpE/d/XuSlircu7dbpA0ULQo5\nIBh/d/e/uHuLuze6+1J3f8bdd7j765JmSvp8J8+f6+7L3L1J0t2SRuYkagAAcuMKSZPdfYO7b5P0\nI0lfNTNTuKgbIOmzkby51N0/DjJYIAjMkQOC8WZsw8wOknSLpCMl7aLwv81nOnn+2zHHn0jaLdmF\nAAAUkkixtp+kB83MYx7qJamfpN9I2kfSXDPbTdJvJf3Q3ZtzHiwQIHrkgGB4u/Ydkl6QdIC77yFp\nqiTLeVQAAATM3V1SSNLJ7l4Z89XH3d919+3uPtXdD5J0oqTzJV3Q+vSg4gZyjUIOyA+7S/pA0sdm\ndrA6nx8HAECx+7Wk6Wa2nySZ2UAz+1Lk+FQzO8TMekn6UOF55i2R522U9JkgAgZyjUIOyA/fkzRe\n0kcK987NCTYcAAACdZOkRyUtiqzw/JSkIyKPVUu6X+Gc+YKkBxXNm7dKutjM3jezm3IbMpBbFu69\nBgAAAAAUCnrkAAAAAKDAUMgBAAAAQIGhkAMAAACAAkMhBwAAAAAFhkIOAAAAAArMTkEH0Jn+/fv7\nkCFDgg4DAJBlzz333LvuPiDoOAoF+REASkeyHJnXhdyQIUO0bNmyoMMAAGSZma0LOoZCQn4EgNKR\nLEcytBIAAAAACgyFHAAAAAAUGAo5AAAAACgwFHIAAAAAUGAo5AAAAACgwFDIAQAAAECByUghZ2Z3\nmtk7ZvZCksdPMrMPzOz5yNfUTNwXAAAAAEpRpvaRmy3pF5J+28k1T7r72Rm6HwAAAACUrIwUcu6+\n2MyGZOK1AAD5r74hpBkLV2vDlkZVVVZoUu0w1Y2qDjosAABKRi7nyB1rZv80s7+a2aE5vC8AIIPq\nG0KaMm+lQlsa5ZJCWxo1Zd5K1TeEgg4NAIDgffqptH591m+Tq0JuuaT93f1wST+XVJ/sQjObaGbL\nzGzZpk2bchQeACBVMxauVmNTc9y5xqZmzVi4OqCIAADIEy+/LB17rPS//5v1W+WkkHP3D919a+T4\nQUnlZtY/ybUz3b3G3WsGDBiQi/AAAN2wYUtjt84DAFD03KVf/lLbjz1OMwafqKHlJ2vM9EVZHa2S\nk0LOzPYxM4scHxW57+Zc3BsAkFlVlRUa++LftPbGs/WF15bGnQcAoOS8/bZ01ll6//aZqrvwJt0+\n7FS5WdanHmRksRMz+4OkkyT1N7P1kq6VVC5J7v5rSV+W9P+Z2Q5JjZIucHfPxL0BADn0ySd68r9P\nV6/mZi2rPliPf+ZISVJFeZkm1Q4LODgAAHKsvl664grpW9/S2DFX6Y2PmuIebp16kI0FwTK1auWF\nXTz+C4W3JwAAFKrZs6VvflO9JD0x+3794K3dpC2NqmbVSgBAqdm6VfrP/5Qee0yaN0867ji9OXlB\nwkuzNfUgU/vIAQAKUErbCGzdKu2+e/j45JOlRx/V5820JPfhAgAQvKeflr7xDemEE6Tnn5f22ENS\neIpBKEHRlq2pB7ncfgAAkEdS2kbgm9+MFnFLl4Y/eQxPeQYAoLQ0NUnXXSfV1Uk33ijdeWdbESdJ\nk2qHqaK8LO4p2Zx6QI8cAJSozrYRqBvg0n77hU/26ydt2kQBBwAoXa++Gu6F23NPaflyqaqqwyWt\nI1q6HOmSIRRyAFCiko3Zv2vGeGlKZCPT2bOl8eNzFxQAAPnEXZo1S5oyRbr2Wunf/k3qlXxQY92o\n6pzNGaeQA4AS1X4s/6APNurvv74sekFLC71wAIDS9c470re+Jb3xhrR4sXTIIUFHFIc5cgBQomLH\n8j/1y0vairil034puav++Q0aM32Rhk5ekPVNTQEAyCsLFkgjR0oHHyw980zeFXESPXIAULLqRlWr\n74rn9PlLxradq1++XnWjqtsWQmmdQ9e6EErr8wAAKEqffCJ9//vhQu4Pf5A+//mgI0qKHjkAKFVm\n0SJu1izJPW6idrKFUAAAKErLlkmjRkkffSStWJHXRZxEjxwAlJ4nnpBOOinadu9wSbKFULK1qSkA\nAKlKaQ/U7mhsDO+T+tpr0s9/Ln31q5kLNovokQOAUmIWLeJ+9rOERZyUfPPSbG1qCgBAKlLaA7U7\nZs2SdtklvMn38uUFU8RJFHIAUBoWLoxfgdJd+s53kl6e601NAQBIRcaG/ruH8+K3vhVuf/qpNGhQ\nhqLMDQo5ACgC9Q2h5CtMmklnnBE+/s1vkvbCxaobVa1p44arurJCJqm6skLTxg1noRMAQKAyMvT/\nH/+I7gU3eXI4L5aXZyC63GKOHAAUuGQrTA794106fPo10QtTKOBi5XJTUwAAUtF+D9TY8yk5/PDw\nQiaStHGjNHBgBqPLLXrkAKBAJOt1SzTMZNX1Z0aLuOnTu13EAQCQj3o89H/DhvAIlRUrpKOPDufF\nAi7iJHrkAKAgdLavW+xwksufmaspj8+OPpECDgBQRGK3yUl51cqrrpJuuy18vGyZdOSROYg0+yjk\nACCPJFtSubPJ3a3DTNbeeHbbYz887QotOvl8Lcn1G0AHZnaGpNsklUma5e7T2z1+iaQZklonNv7C\n3WflNEgAKCApD/3fvl3q0yfabmmJX/irwFHIAUCeSLXXLdaGLY1avPJO7ffgvLZzQ65+QBXlZZrG\nCpOBM7MySbdLOk3SeklLzWy+u7/U7tI57n5lzgMEgGJ1993S178ePv7tb6VvfCPYeLKAQg4A8kQq\nvW7t/SumF+6XZ0zQjMPrVJ2JzVGRKUdJWuPur0uSmd0raayk9oUcACAT3KMrUkrStm1S797BxZNF\nFHIAkCc663W79asj43rrfjP3RzrltaXRi9z1bUnfzkGc6JZqSW/GtNdLOjrBdeeZ2YmSXpH0n+7+\nZoJrAACdWb48Ov/tO9+RfvazYOPJMlatBIA8kWzp5KrKirh93dbeeHa0iLv9dhY0KXx/kTTE3UdI\nekTSXYkuMrOJZrbMzJZt2rQppwECQN47/vhoERcKFX0RJ1HIAUDe6GpJ5brTDteSKadEH3SXvk0f\nXJ4LSdovpj1I0UVNJEnuvtndt0easyQlXE7N3We6e4271wwYMCArwQJAwdm0KbyAyZIl0qGHhnNj\nVVXQUeUEhRwA5InYXjeTVF1ZoWnjhofnuplJmzeHL5w1i164wrFU0oFmNtTMdpZ0gaT5sReY2b4x\nzXMkrcphfABQuK65JroX3JIl0gsvBBtPjjFHDgDySIclldsvk0wBV1DcfYeZXSlpocLbD9zp7i+a\n2Y8lLXP3+ZL+3czOkbRD0nuSLgksYAAoBNu2SRUx0xGKbFuBVNEjBwD5yD0+Kd15J0VcgXL3B939\nc+7+WXf/SeTc1EgRJ3ef4u6Huvvh7v4Fd3852IgBII9dfXW0iJs6tWO+7Kb6hpDGTF+koZMXaMz0\nRapvCHX9pDxBjxwAdCLZBt1ZRS8cAAAdxebH99+XKivTernO9m8thC186JEDgCRa/4MPbWmUK/of\nfNY+rWs/NGTuXIo4AAD++tf4/OiedhEndb5/ayHISI+cmd0p6WxJ77j7YQkeN0m3SfqipE8kXeLu\nyzNxbwDIls7+g8/4J3VZ7IULpFcRAIBMiM2PS5dKNTUZe+nO9m8tBJnqkZst6YxOHj9T0oGRr4mS\nfpWh+wJA1uTkP/gdO+KT1J//nPEiLqe9igAAZMKbb3bshctgESd1vn9rIchIIefuixVeaSuZsZJ+\n62FPS6pst9wyAOSdrP8HbyaVl0fb7tK4cZl57YhCHzYCAChBNTXS4MHh41/8ImvTDLravzXf5WqO\nXLWkN2Pa6yPnACBvZe0/+MbG+E8ZH3ooa0mq0IeNAABKSFNTOD8+91y43dws/du/Ze12ne7fWgDy\nbtVKM5uo8PBLDW6txAEgAK3/kWd0flmOV6SsqqxQKEHRVijDRgAAJeL666Uf/jB8fPrp0sKFOblt\nh/1bC0iuCrmQpP1i2oMi5zpw95mSZkpSTU0Ny7UBCFTG/oP/4IP4FbYee0w6+eT0X7cLk2qHxS2t\nLBXWsBEAQAmI/ZDznXekAQOCi6WA5KqQmy/pSjO7V9LRkj5w97dydG8ACFaA+8JlpVcRAIBMeOIJ\n6aSTom223OmWTG0/8AdJJ0nqb2brJV0rqVyS3P3Xkh5UeOuBNQpvP/DNTNwXAFIVyBL8mzZJAwdG\n288+K40end17JlDIw0YAAEUq9kPOJ5+Ujj8+uFgKVEYKOXe/sIvHXVL2ZioCQCdal+BvHV7YugS/\npOwVOAH2wgEAkLc2bpT22SfaJj/2WK5WrQSAwOR0Cf5//Su+iHvhBZIUAACSdMop0SJu2jTyY5ry\nbtVKAMi0nC3BTy8cAAAdNTdLO8WUHU1N8W30CD1yAIpe1jf2fvHF+CLu1Vcp4gAAkKSf/jRatB19\ndDg/UsRlBN9FAEUvq0vw0wsHAEBisTly/XqpmoW3MokeOQBFr25UtaaNG67qygqZpOrKCk0bNzy9\nhU4aGjomKIo4AADCqzTH5kh3irgsoEcOQEnI6BL89MIBAJBYbI5cuFA6/fTgYily9MgBQKqefDI+\nQb33HkUcAABSOCe274WjiMsqCjkASIWZdOKJ0ba71LdvcPEAAJAvzjtP6tcvfPyDH/AhZ44wtBIA\nOjN3rnT++dH21q3SrrsGFw8AAPnCXeoV0y+0bZvUu3dw8ZQYCjkASCaNuXD1DSHNWLhaG7Y0qqqy\nQpNqh2Vujh4AAEGbNUv61rfCx5/7nLR6dbDxlCAKOQBoLzY5Sd3+hLG+IRS33UFoS6OmzFspKbzo\nCkUeAKCgxX7Q+frr0tChwcVSwijkACBWBlaknLFwddyedZLU2NSsGQvDn1Z2VuQBAJC3VqyQDj88\n2mYuXKBY7AQAJOmmm+KLuB07epygNmxpTHq+qyIPAIC81Lt3tIi77z6KuDxAjxwAZHhfuKrKCoUS\nFHNVlRWdFnkAAOSdjz6S9tgj2qaAyxv0yAEoXVdfHV/EtbRkJEFNqh2mivKyuHMV5WWaVDtMVZUV\nCZ+T7DwAAIG59NJoEXfllRRxeYYeOQClKcO9cLFa57olW9Akdo6cFC3yAADIC+23Ffj4Y2mXXYKL\nBwlRyAEoKGmv+PjNb0qzZ0fbWfp0sW5UdcK4uiryAAAI1L33ShdeGD4eMEB6551g40FSFHIACkZX\ny/p3KYu9cN2RrMgDACBQsXly1SrpoIOCiwVdYo4cgILR4xUfzzorPjm5M84fAIBWr7zSMU9SxOU9\neuQAFIwerfiYxV44NvYGABS8ffeV3n47fPz730sXXRRsPEgZhRyAgtHZsv4dHHmktHx5tJ3hHri0\nh3kCABCkxsb4BUxaWjp++Im8xtBKAAWjs2X945hltYiT0hjmCQBA0P7zP6NF3De+Ec6TFHEFhx45\nAAWjyxUfq6ulDRuiT8jiPDg29gYAFKTYgu2DD+I3+0ZBoZADUFCSrviY4xUpuzXMEwCAoM2fL40d\nGz7u1Utqbu78euQ9hlYCKGxmgaxImfIwTwAAgmYWLeIaGijiigQ9cgAKV2wB16dPeOJ2jrCxNwAg\n761bJw0ZEm2z9U5RyUghZ2ZnSLpNUpmkWe4+vd3jl0iaISkUOfULd5+ViXsDKEFs7A0AQOcOPVR6\n6aXw8R13SBMnBhsPMi7tQs7MyiTdLuk0SeslLTWz+e7+UrtL57j7leneD0BhyNoea7FF3Gc/K61Z\nk/5rAgBQLD79VOrdO9pubg7PiUPRycRP9ShJa9z9dXf/VNK9ksZm4HUBFKjWPdZCWxrliu6xVt8Q\n6vK5SSWaC0cRBwBA1NSp0SLunHPCuZIirmhl4idbLenNmPb6yLn2zjOzFWY218z2y8B9AeSpjO+x\nFlvAHX88Y/wBAGjPTPqf/wkfv/uudP/9wcaDrMtVif4XSUPcfYSkRyTdlexCM5toZsvMbNmmTZty\nFB6ATMrYHmuJeuGefDKNyAAAKDKPPtoxV/brF1w8yJlMFHIhSbE9bIMUXdREkuTum919e6Q5S9KR\nyV7M3We6e4271wwYMCAD4QHItWR7qXVrj7XYpDRuHL1wAAC0Zyaddlr4+B//IFeWmEwUckslHWhm\nQ81sZ0kXSJofe4GZ7RvTPEfSqgzcF0CeSmuPtUS9cH/+c4YjBHLHzM4ws9VmtsbMJid4vLeZzYk8\n/oyZDcl9lAAKyoYNHXPlMccEFw8CkXYh5+47JF0paaHCBdof3f1FM/uxmZ0TuezfzexFM/unpH+X\ndEm69wWQv+pGVWvauOGqrqyQSaqurNC0ccM7X7XSPT4pXXEFnyyi4MWs7HympEMkXWhmh7S77DJJ\n77v7AZJulXRjbqMEUFDGjJGqI/n0llvIlSUsI/vIufuDkh5sd25qzPEUSVMycS8AudXTbQS6tcda\nnuwLB2RB28rOkmRmrSs7x27RM1bSdZHjuZJ+YWbmzj8EADF27JDKy+PbZWXJr0fRYz1SAEllZRuB\nWC0t8UXcD39IEYdik8rKzm3XREa5fCCJlQoARM2YES3iTjwxnCsp4kpeRnrkABSnzrYRSHtzb3rh\ngG4xs4mSJkrS4MGDA44GQM7E5su33pL22Se4WJBX6JEDkFTGthGI1dQUn5RuvZUiDsWsy5WdY68x\ns50k7Slpc/sXYlVnoMQ89VTHBU0o4hCDHjkASVVVViiUoGjr1jYCseiFQ+lpW9lZ4YLtAklfa3fN\nfEnjJf1D0pclLWJ+HFDiYvPlY49JJ58cXCzIW/TIAUgqrW0EYjU2xiel2bMp4lASUlzZ+TeS+pnZ\nGknfldRhiwIAJeLddzv2wlHEIQl65AAk1ToPrierVrahFw4lLoWVnbdJOj/XcQHIM2edJT0Y+a/i\nuuuka68NNBzkPwo5AJ3q1jYCsT76SNpjj2i7vl4aOzZzgQEAUAxaWuJXoNy+Xdp55+DiQcFgaCWA\nzDOLL+LcKeIAAGjvV7+KFnEjRoTzJUUcUkSPHIDM2bxZ6t8/2l6yRDruuODiAQAgX8VOPVi3TmJb\nEXQThRyAzGAuHAAAXVu+XDryyGibfIkeYmglgPSEQvFF3D//SVICACARs2gR98AD5EukhR45AD3X\nrheufvl61Y3owcIoAAAUsw8+kCoro20KOGQAPXIAum/Nmrgi7sSJ/09Drn5AU+atVH1DKMDAAADI\nMxddFC3ivvc9ijhkDD1yALqnXS/ckKsfaDtubGrWjIWre7ZdAQAAxcRd6hXTZ/LJJ1JFRXDxoOhQ\nyAFIzT//KY0c2dY8+tt3aePu/TpctmFLY0ovV98QSm+jcQAA8tXvfiddfHH4eNAg6c03g40HRYlC\nDkDXEqxIudP0RVKCoq2qsutPG+sbQpoyb6Uam5olSaEtjZoyb6UkUcwBAApbbM585RXpwAODiwVF\njTlyQBGpbwhpzPRFGjp5gcZMX5TyfLWkz1u0KD4hvftu29j+SbXDVFFeFvc6FeVlmlQ7rMv7zVi4\nuq2Ia9U6LBMAgIL00kvxOdOdIg5ZRY8cUCR62suV7Hl1RwyKv7Dd5OzW1+zJ8Mhkwy9THZYJAEBe\n2Wsv6f33w8dz5khf+Uqw8aAkUMgBRaKzXq7Oiqv2zzv9lX9o5n0/iV7wwQfSHnskfG7dqOoeDYWs\nqqxQqIfDMgEAyBsffyzttlu03dLScToCkCUMrQSKRE97uWIfX3vj2fFFnHvSIi4d6QzLBAAgL3z7\n29EibsKEcM6kiEMO0SMHFIme9nJVVVbo2Cf/opsf/GnbuWHf/bP6D6jUkoxHGZbOsEwAAAIXW7B9\n9FF8rxyQIxRyQJGYVDssbq6blFov15Ipp8S1h1z9QE56x3o6LBMAgMDMmyedd174eNddpa1bg40H\nJY1CDigS3e7luvVW6bvfbWueeP3DevOjT1VN7xgAAB3F9sKtXCkddlhwsQCikAOKSsq9XAn2hVuc\nnZAAAChsr70mHXBAtN1uFWcgKCx2AgSkp3u+peW//zu+iGtuJiEBAJDMZz8bLeLuvJOcibxCjxwQ\ngJ7u+ZaWBL1wAAAggW3bpIqYxcLYVgB5KCM9cmZ2hpmtNrM1ZjY5weO9zWxO5PFnzGxIJu4LFKrO\n9nzLuMsvj08+LS0UcQAAJDN5crSIO/98thVA3kq7R87MyiTdLuk0SeslLTWz+e7+Usxll0l6390P\nMLMLJN0o6avp3hsoVD3d863b6IUDACB1sXnzvfekvn2DiwXoQiZ65I6StMbdX3f3TyXdK2lsu2vG\nSrorcjxX0ilmfLSB0pVsb7eu9nxL2bhx8cnInSIOAIBkHnqoY96kiEOey0QhVy3pzZj2+si5hNe4\n+w5JH0jql4F7AwVpUu0wVZSXxZ3L2N5tZtJ990XbFHAAACRnJp15Zvh46VLyJgpG3q1aaWYTzWyZ\nmS3btGlT0OEAWVE3qlrTxg1XdWWFTFJ1ZYWmjRue3kInxx1HLxwAAKlav75j3qypCS4eoJsysWpl\nSNJ+Me1BkXOJrllvZjtJ2lPS5kQv5u4zJc2UpJqaGn4LRdFKec+3VDAXDgCA1NXUSM89Fz7+2c+k\n73wn2HiAHshEIbdU0oFmNlThgu0CSV9rd818SeMl/UPSlyUtcuc3TSBtQ4dKa9dG2/yzAgAguaYm\naeedo+3mZqlX3g1QA1KS9t/cyJy3KyUtlLRK0h/d/UUz+7GZnRO57DeS+pnZGknfldRhiwIA3WRG\nEQcAQKp+8pNoEXfaaeG8SRGHApaRDcHd/UFJD7Y7NzXmeJuk8zNxL6DkMYwSAIDuic2dGzdKAwcG\nFwuQIXwMARQSijgAAFL3xBMdFzShiEORyEiPHIAso4ADAKB7YnPn4sXSCScEFwuQBRRyQL6LTUS9\neql+2RuaMX2RNmxpVFVlhSbVDsvc6pcAABS6d96R9t472ubDTxQphlYC+cqsw3CQ+mVvaMq8lQpt\naZRLCm1p1JR5K1Xf0H7HDwAAStCpp0aLuBtuoIhDUaNHDshHsQXc4MHSunWSpBkLV6uxqTnu0sam\nZs1YuJpeOQBA6WpulnaK+bW2qSm+DRQheuSAfJKgF661iJOkDVsaEz4t2XkAAIrez34WLdpGjw7n\nToo4lAD+lgP5IraAO+YY6R//6HBJVWWFQgmKtqrKimxGBgBAforNnW++KQ0aFFwsQI7RIwcELVEv\nXIIiTpIm1Q5TRXlZ3LmK8jJNqh2WzQgBAMgvzz7bMXdSxKHEUMgBQXGPT0LnntvlpOy6UdWaNm64\nqisrZJKqKys0bdxw5scBAEqHmXT00eHjhx5iQROULIZWAkFIY1+4ulHVFG4AgNLz/vvSXntF2xRw\nKHH0yAG51L4XbvJkEhEAAF358pejRdyUKeROQPTIAbmTRi8cAAAlyV3qFdPvsG2b1Lt3cPEAeYQe\nOSDbduyIL+JmzKCIAwCgK7/5TbSIO+CAcO6kiAPa0CMHZBO9cAAAdF9s/nztNekznwkuFiBP0SMH\nZMO2bfFJaNYsijigxJjZXmb2iJm9Gvmzb5Lrms3s+cjX/FzHCeSVlSs7bitAEQckRI8cSkp9Q0gz\nFq7Whi2Nqqqs0KTaYZlfAZJeOABhkyU95u7TzWxypH11gusa3X1kbkMD8lCfPtL27eHjefPC2/IA\nSIoeOZSM+oaQpsxbqdCWRrmk0JZGTZm3UvUNoczc4KOP4ou4uXMp4oDSNlbSXZHjuyTVBRgLkL9a\n82drEedOEQekgEIOJWPGwtVqbGqOO9fY1KwZC1d3+dz6hpDGTF+koZMXaMz0RR2LPzNpjz2ibXfp\nvPMyETaAwrW3u78VOX5b0t5JrutjZsvM7Gkzo9hDabnssmj+/Pa3+QAU6AaGVqJkbNjS2K3zrVp7\n8lqLwNaePEmqG7SzNHBg9OJHHpFOPTUzAQPIe2b2qKR9Ejx0TWzD3d3Mkv2Gur+7h8zsM5IWmdlK\nd38twb0mSpooSYMHD04zciBg7bcV2LpV2nXX4OIBChCFHEpGVWWFQgmKtqrKik6fl6wnr+6IQfEX\n8ikiUHLcPeknN2a20cz2dfe3zGxfSe8keY1Q5M/XzexxSaMkdSjk3H2mpJmSVFNTw384KFxz5kgX\nXBA+3msvafPmYOMBChRDK1EyJtUOU0V5Wdy5ivIyTaod1unz2vfY7fvhJq298ezoiSVLKOIAJDJf\n0vjI8XhJ97e/wMz6mlnvyHF/SWMkvZSzCIFcM4sWcS+9RBEHpIEeOZSM1tUpu7tqZWxPXlwBJ1HA\nAejMdEl/NLPLJK2T9BVJMrMaSVe4+wRJB0u6w8xaFP5wdbq7U8ih+LzyijQs5oNT8ieQNgo5lJS6\nUdXd3m5gUu0w/XLWQj38y8vazp0z8Ze69IpzWIIOQFLuvlnSKQnOL5M0IXL8lKThOQ4NyK2qKumt\nyLo/v/ud9PWvBxsPUCQo5L7CxXsAACAASURBVIAu1B0xKK5gGzPtsezsPwcAQDFpbJR22SXabmnp\nuNcqgB5jjhyQzMqV8Qln7VrJXUsmn0wRBwBAZ7773WgR9/Wvh4dSUsQBGUWPHJBI+2TDWH4AAFIT\nm0O3bJH23DO4WIAillaPnJntZWaPmNmrkT/7Jrmu2cyej3zNT+eeQFY99VR8Atq4kSIOAIBU/OUv\n8TnUnSIOyKJ0e+QmS3rM3aeb2eRI++oE1zW6+8g07wVkF71wAAD0TGwOXb5cGjUquFiAEpHuHLmx\nku6KHN8lsYgfCtBDD3UcBkIRBwBA19at69gLRxEH5ES6hdze7h5ZT1ZvS9o7yXV9zGyZmT1tZhR7\nyB9m0plnRtsMAwEAIDWHHSYNGRI+/tWv+BAUyLEuh1aa2aOS9knw0DWxDXd3M0v2L3h/dw+Z2Wck\nLTKzle7+WpL7TZQ0UZIGDx7cVXhAz8yZI11wQbT9ySdSRUVw8QAAUCg+/VTq3Tvabm6WerEQOpBr\nXf6rc/dT3f2wBF/3S9poZvtKUuTPd5K8Rijy5+uSHpeUtM/d3We6e4271wwYMKAHbwnogll8EedO\nEQcAQCquvTZaxJ19djiHUsQBgUj3X958SeMjx+Ml3d/+AjPra2a9I8f9JY2R9FKa9wW6b+bM+HH8\nn37KMBAAAFJlJv34x+HjTZvCq1QCCEy6q1ZOl/RHM7tM0jpJX5EkM6uRdIW7T5B0sKQ7zKxF4cJx\nurtTyCG3WJESAICeeewx6dRTo21yKJAX0irk3H2zpFMSnF8maULk+ClJw9O5D9BjN94oTZ4cbWdh\nHH99Q0gzFq7Whi2Nqqqs0KTaYaobVZ3RewAAEIjYD0Kfeko69tjgYgEQJ90eOSB/5aAXrr4hpCnz\nVqqxqVmSFNrSqCnzVkoSxRwAoHC99ZZUVRVt0wsH5B1mp6L4TJoUX8S1tGQtAc1YuLqtiGvV2NSs\nGQtXZ+V+AABk3QknRIu4GTMo4oA8RY8cikuO58Jt2NLYrfMAAOStHTuk8vL4dllZcPEA6BQ9cigO\nl14aX8S55+QTxKrKxNsWJDsPAEBeuvnmaBF3/PHhHEoRB+Q1euRQ+AJckXJS7bC4OXKSVFFepkm1\nw3IWAwAAaYnNoxs2SPvuG1wsAFJGjxwK13nnBdILF6tuVLWmjRuu6soKmaTqygpNGzechU4AAPnv\nqac65lGKOKBg0COHwpRH+8LVjaqmcAMAFJbYPPrYY9LJJwcXC4AeoUcOheXsswPvhQMAoGC9+27H\nPEoRBxQkCjkUDjNpwYLw8W67UcABANAdX/qSNGBA+HjqVPIoUOAYWon8d8wx0jPPRNskHgAAUtfS\nEr8C5fbt0s47BxcPgIygRw55q74hFO6FixRxW/cbQhEHAEAS9Q0hjZm+SEMnL9CY6YvCefTXv44W\ncYceGs6jFHFAUaBHDnnp3SOOUV1DtBduyNUPqKK8TNMaQiwsAgBAO/UNobjtcEJbGlV3xKDoBWvX\nSvvvH0xwALKCHjnkHzP1jxRxDxx0goZc/YAkqbGpWTMWrg4yMgAA8tKMhavbirhDN76mtTeeHX3Q\nnSIOKEL0yCF/HHyw9PLLbc3WAi7Whi2NuYwIAICC0JofYwu4S8+bqr8dcJT+FVRQALKKQg75IXYp\n5Cuu0Jj9z5cSFG1VlRU5DAoAgMJwYJ9mPfyjsW3t1g9Dq8mbQNFiaCWC1a9fx/1sfvUrTaodpory\nsrhLK8rLNKl2WI4DBAAgz110UVsR9/9G17UVceRNoLjRI4eMqG8IacbC1dqwpVFVlRWaVDus60VJ\nYgu4KVOkG25oa7Y+t9uvCQBAqXCXekU/k5//1Kua/cQbMvImUBIo5JC2RCtlTZm3UpISJ5DYAk5K\nuqVA3ahqEhAAAIn87nfSxReHj6uqpFBI50g659gDAg0LQO4wtBJpi10pq1XSFSZji7ibbmJfOAAA\nusssWsS98ooUCgUbD4BA0COHtCVbSTLufIq9cAAAIIlVq6RDDom2yaVASaNHDmlLtpJkVWVFOMnE\nFnF33EHiAQCgu/r1ixZx995LLgVAjxzSN6l2WNwcOSm8UtaSKadIU2IuJOkAANA9H38s7bZbtN3S\n0nGUC4CSRI8cUlbfENKY6Ys0dPICjZm+SPUN4TH5daOqNW3ccFVXVsgkDdqjt1Zdf2b0iffcQxEH\nAEB3ffvb0SLusss6jnIBUNLokUNKulqZsm2FSebCAQCQvth8+uGH0u67BxcLgLxEjxxS0uXKlE1N\n8UnngQco4gAA6K5586L5tE+fcC6liAOQAD1ySEmnK1MG0AvXow3IAQDIZ7H5dMUKafjw4GIBkPfS\n6pEzs/PN7EUzazGzmk6uO8PMVpvZGjObnM49EYxEK1P23vGp/nXj2dETjz+esyJuyryVCm1plCs6\nzLN1zh4AAAXl9dfjizh3ijgAXUp3aOULksZJWpzsAjMrk3S7pDMlHSLpQjM7JNn1yE+Taoeporys\nrb32xrO1+pZx0Qvcpc9/PiexdGsDcgAA8tkBB0if/Wz4eNYspiUASFlaQyvdfZUkWecrKB0laY27\nvx659l5JYyW9lM690blMDz1sfe7t8xv0yHVfij7w7LPS6NHphtstKW1ADgBAPtu+PTwHrhXbCgDo\nplzMkauW9GZMe72ko3Nw36KXrFjraoXJnqo7YpDqYk8E9KlhVWWFQgmKtmQbkwMAkFemTJGmTw8f\nf/nL0p/+FGw8AApSl4WcmT0qaZ8ED13j7vdnOiAzmyhpoiQNHjw40y9fNDor1jobetijQm7LFqlv\n32j71VfDQ0ECkmwD8km1wwKLCQCAlMT2ur33Xnx+BYBu6HKOnLuf6u6HJfhKtYgLSdovpj0oci7Z\n/Wa6e4271wwYMCDFW5Sezoq1jA49NItPMu6BFnFSxw3IqysrNG3ccFatBJBXWBAMcR56qOOCJhRx\nANKQi6GVSyUdaGZDFS7gLpD0tRzct6h1VqxlZOjhpk3SwIHR9ptvSoMGdTfMrGnbgBwA8lfrgmB3\nJLsgZkGw0xSeerDUzOa7O/PIi0lsARfA3HIAxSnd7QfONbP1ko6VtMDMFkbOV5nZg5Lk7jskXSlp\noaRVkv7o7i+mFzaSFWWtc+ViV5iUujn00Cy+iHPPqyIOAAqBu69y966W021bEMzdP5XUuiAYisH6\n9R174SjiAGRIWoWcu9/n7oPcvbe77+3utZHzG9z9izHXPejun3P3z7r7T9INGh23A5CixVqPhx62\nSzhnXTNX9cvXZyF6AEBEogXBGG5QDEaPlvaLzCz56U/ZVgBAxuViaCWyoLUoS7bFQLeHHrZb8njI\n1Q9IO5SR1S4BoFjlckEwFgMrEE1N0s47R9s7dkhlZcmvB4AeopArYBmZJ/baa3GLl4z4j3v1YZ/d\n2tpprXYJAEXO3U9N8yVSXhDM3WdKmilJNTU1dO/koxtukK65Jnx86qnSI48EGw+AokYhV8ra9cIN\nvfoBJfrNgI22ASBrWBCsWMTm1I0b4+eaA0AWpDVHDgXqhRfiE87WrZJ7pwuoAAC6hwXBSsTixR0X\nNKGIA5AD9MiVmna9cLGTr9loGwAyx93vk3RfgvMbJMUtCCbpwRyGhkyJzalPPCGdeGJwsQAoORRy\npWLpUumoo6Ltbduk3r3jLulqARUAACDpnXekvfeOtlmREkAAKORKQSe9cO2x0TYAAJ047TTp0UfD\nxz/5ifSDHwQbD4CSRSFXzJ54QjrppGi7qUnaiR85AADd1twcn0M//VQqLw8uHgAlj8VOipVZfBHn\nThEHAEBP/Pzn0RxaUxPOqRRxAALGb/bF5skn4ydbNzdLvajXAQDokdjpCW++KQ0aFFwsABCD3/CL\niVm0iDvooPAnhhRxAAB037PPdtxWgCIOQB7ht/xisHBhfLJpaZFWrQouHgAACpmZdPTR4eOHHmJV\nSgB5iaGVhS62gDvuOGnJkuBiAQCgkL3/vrTXXtE2BRyAPEaPXKF64omOQz4o4gAA6Jmrr44WcVdf\nTREHIO/RI1eIYgu4731Puvnm4GIBAKCQtZ9Pvn27tPPOwcUDACmiR66QPPRQx144ijgAAHpm/vxo\nEVdXF86rFHEACgQ9coUitoCbOlX60Y+CiwUAgELHtgIAChyFXL6bN08677xomzH7AIAiV98Q0oyF\nq7VhS6OqKis0qXaY6kZVZ+bF16yRDjww2iavAihQDK3MZ2bRIu6mm0g2AICiV98Q0pR5KxXa0iiX\nFNrSqCnzVqq+IZT+i48ZEy3iFi0irwIoaPTI5aPf/176xjeibRINAKBEzFi4Wo1NzXHnGpuaNWPh\n6p73yn3yibTrrtE2eRVAEaBHLt+YRYu4X/6SZAMAKCkbtjR263yX/ud/okXctGnkVQBFgx65fDFz\npnT55dE2iQYAUIKqKisUSlC0VVVWdO+F2m8r8MknUkU3XwMA8hg9cvnALFrEzZ5NEQcAKFmTaoep\norws7lxFeZkm1Q5L/UUeeSRaxJ10UjivUsQBKDL0yAXpvvukceOibQo4AECJa50H1+NVK2O3FXj9\ndWno0JSeltWVMgEgCyjkghKbaJ58Ujr++OBiAQAgj9SNqu5+EfXGG9L++0fb3fhwtHWlzNZFVlpX\nymyNBQDyEUMrc+2ee+KLOHeKOAAA0nHWWdEibsGCbo9w6WylTADIV2n1yJnZ+ZKuk3SwpKPcfVmS\n69ZK+khSs6Qd7l6Tzn0LVmwB9+yz0ujRwcUCAECh275d6tMn2m5pic+1Kcr4SpkAkAPp9si9IGmc\npMUpXPsFdx9ZkkXcnXd27IWjiAMAoOf+93+jRdx//3c4t/agiJOSr4jZ7ZUyASCH0uqRc/dVkmQ9\n/I+zJMR+b55/Xjr88OBiAQCgGMTm1o8+knbbLa2Xm1Q7LG6OnNSDlTIBIMdyNUfOJT1sZs+Z2cQc\n3TNYt9/esReOIg4AgJ578slobh05Mpxb0yzipPCCJtPGDVd1ZYVMUnVlhaaNG85CJwDyWpc9cmb2\nqKR9Ejx0jbvfn+J9jnf3kJkNlPSImb3s7gmHY0YKvYmSNHjw4BRfPo+034D0pZekgw8OLh4AAIrB\nTjtJzZEes1WrpIMOyujL92ilTAAIUJeFnLufmu5N3D0U+fMdM7tP0lFKMq/O3WdKmilJNTU1hbWx\n2owZ0n/9V7TNvnAAAKTn7belffeNtsmtACApB0MrzWxXM9u99VjS6QovklI8WidYtxZxr77a7f1r\nxkxfpKGTF2jM9EWqbwhlKVAAAArIV78aLeLmzqWIA4AY6W4/cK6kn0saIGmBmT3v7rVmViVplrt/\nUdLeku6LLIiyk6R73P2hNOPOHz/+sXTttdF2N5MMm5ACANBOc3N4KGVsuxdb3wJArLT+V3T3+9x9\nkLv3dve93b02cn5DpIiTu7/u7odHvg51959kIvDAte5V01rErVvXo08K2YQUAIAYy5ZFi7jvf7/j\n3HMAgKTcrVpZXCZPlsrKwse77x5OMj1cmIVNSAEAiPja18L7rJpJjY3huecAgITSGlpZctoP9QiF\npKqqtF6yqrJCoQRFG5uQAgBKRigkDRoUPr7tNunf/z3YeACgANAjl6orr4wWcVVV4V64NIs4KbwJ\naUV5Wdw5NiEFAJSMG2+MFnFvv00RBwApokeuK01N0s47R9sbN0oDB2bs5VsXNJmxcLU2bGlUVWWF\nJtUOY6ETAEBx++QTadddw8eXXir95jfBxgMABYZCrjPf/KY0e3b4+KCDwhuQZgGbkAIASsr8+dLY\nseHj55+XDj882HgAoABRyCWyY4dUXh5tv/ee1LdvcPEAAFAMWlqkkSOllSulESOkhgZWpASAHirZ\nQq6+IZR4OOMdd0hXXBG+aPRo6dlngw0UQMFqamrS+vXrtW3btqBDyRt9+vTRoEGDVB77YRlKwz//\nGS7iJOn++6Vzzgk2HgB5g3wZ1t0cWZKFXKJNuKf+abnqjhgUveiTT6QKVo4E0HPr16/X7rvvriFD\nhsjMgg4ncO6uzZs3a/369Ro6dGjQ4SCXLrtMuvPO8PHHH0u77BJsPADyCvmyZzmyJMcztN+Ee/xz\nf9GKaZFPBm+5JbwiJUUcgDRt27ZN/fr1K9mk1J6ZqV+/fiX/iWtJ2bgxvCfcnXeGV6d0p4gD0AH5\nsmc5siR75Fo32955R5NeueXctvPDvnefVn+3LvmwSwDoplJOSonw/Sght90mXXVV+Hj9eqmaPAog\nOfJD978HJdkjV1VZoc9sXt9WxP3wtCs05OoH1L//Hm3DLkNbGuUKD7ucMm+l6htCwQYNAAEbMmSI\n3n333bSvQZHbti3cC3fVVdIFF4R74SjiAJSIXObK0uuRa27WHW8vUvXdP9fM0efqps+P146yndo2\n4W4/7FKSGpuaNWPhanrlAABFIyujTx56SDrzzPDx0qVSTU36gQIAEiqtHrmXXpKOO06HrXhKS+f8\nVXeNu1LNZTupurJC08YNV92o6rZhl+0lOw8A+Wzt2rU66KCDdMkll+hzn/ucLrroIj366KMaM2aM\nDjzwQD377LN67733VFdXpxEjRuiYY47RihUrJEmbN2/W6aefrkMPPVQTJkyQu7e97u9//3sdddRR\nGjlypC6//HI1NzcnCwF5KOOjT9ylY44JF3EHHBDexociDkCBKNRcWRqF3I4d0rRp0oknhjf5fvRR\nnf6l47Rk8sn61/SztGTyyW2fQlZVJl7kJNl5AMh3a9as0fe+9z29/PLLevnll3XPPffo73//u26+\n+WbdcMMNuvbaazVq1CitWLFCN9xwgy6++GJJ0o9+9CMdf/zxevHFF3XuuefqjTfekCStWrVKc+bM\n0ZIlS/T888+rrKxMd999d5BvMS+Z2flm9qKZtZhZ0qrGzNaa2Uoze97MluUits5Gn3TbSy+F94J7\n5hlpzhzp1VelsrIMRQoAuVGIubL4h1a+8EK4eKuslJ57Ttp//04vn1Q7LG5rAkltwy4BIC3ZmMgd\n88lfMkOHDtXw4cMlSYceeqhOOeUUmZmGDx+utWvXat26dfrzn/8sSTr55JO1efNmffjhh1q8eLHm\nzZsnSTrrrLPUt29fSdJjjz2m5557TqNHj5YkNTY2auDAgZl/b4XvBUnjJN2RwrVfcPecTS7M2OiT\nK6+Ubr89fPzhh9Luu6cZGQAokHxZiLmy+Au5KVOkyy8P72GTwl+K1p45Vq0EkHEpFF3Z0Lt377bj\nXr16tbV79eqlHTt2dHtzbnfX+PHjNW3atIzGWWzcfZWUnyuxVVVWKJSgaEt59Mm770oDBoSPf/Qj\naerUDEYHoOQFkC8LMVcW/9DK+fOlCRO6VdnXjapOOOwSAIrRCSec0Dbc4/HHH1f//v21xx576MQT\nT9Q999wjSfrrX/+q999/X5J0yimnaO7cuXrnnXckSe+9957WrVsXTPDFwSU9bGbPmdnEXNxwUu0w\nVZTHD39MefTJHXdEi7i1ayniAJSEfMyVxd8jl4efhAJAPrnuuut06aWXasSIEdpll1101113SZKu\nvfZaXXjhhTr00EN13HHHafDgwZKkQw45RNdff71OP/10tbS0qLy8XLfffrv272LoejEys0cl7ZPg\noWvc/f4UX+Z4dw+Z2UBJj5jZy+6+OMG9JkqaKKntZ9FTPRp98umn0l57SR9/LNXVSffdl1YMAFBI\n8jFXmgc01CcVNTU1vmxZTuZ9A0DGrVq1SgcffHDQYeSdRN8XM3vO3YtymUMze1zS9929y4RmZtdJ\n2uruN3d2Xc7z46JF0imnhI+XLJGOOy539wZQ9MiXUd3JkcU/tBIAgDxlZrua2e6tx5JOV3iRlPzg\nLp18criIq6qSmpoo4gAgT1DIAQCQBWZ2rpmtl3SspAVmtjByvsrMHoxctrekv5vZPyU9K2mBuz8U\nTMTtvPpqeFuBv/1N+t3vpFBI2qn4Z2QAQKHgf2QAALLA3e+T1GEimbtvkPTFyPHrkg7PcWhdmzRJ\nujkyuvP998Nb+AAA8go9cgAAIOz998OLhN18szR5cnhoJUUcAOQleuQAAIA0e7b0zW+Gj9eskT77\n2UDDAQB0jkIOAIBS1tQkVVdLmzZJp50mLVzI1j0AUAAYWgkAQKn6+9+lnXcOF3GPPy49/DBFHAAk\nsXbt2rbNv3vihhtuyGA0aRZyZjbDzF42sxVmdp+ZJRxIb2ZnmNlqM1tjZpPTuScAILPSSUzHsRR9\nYTvhBGnPPaXt26XPfz7oaAAgrxVVISfpEUmHufsISa9ImtL+AjMrk3S7pDMlHSLpQjM7JM37AkDR\nqW8Iacz0RRo6eYHGTF+k+oZQTu7bWWLasWNHp8996qmnshEScsVd2rIl3CsHAAUi0/ly6tSp+ulP\nf9rWvuaaa3Tbbbd1uG7y5Ml68sknNXLkSN16661qbm7WpEmTNHr0aI0YMUJ33HGHJOmtt97SiSee\nqJEjR+qwww7Tk08+qcmTJ6uxsVEjR47URRddlFa8rdKaI+fuD8c0n5b05QSXHSVpTWSJZZnZvZLG\nSnopnXsDQDGpbwhpyryVamxqliSFtjRqyryVkqS6UdU9es2pU6dqr7320lVXXSUpnJgGDhyo//iP\n/4i7bvLkyVq1apVGjhyp8ePHq2/fvpo3b562bt2q5uZmLViwQGPHjtX777+vpqYmXX/99Ro7dqwk\nabfddtPWrVv1+OOP67rrrlP//v31wgsv6Mgjj9Tvf/97GcP0SkJ9Q0gzFq7Whi2Nqqqs0KTaYT3+\newsAnclGvrz00ks1btw4XXXVVWppadG9996rZ599tsN106dP180336wHHnhAkjRz5kztueeeWrp0\nqbZv364xY8bo9NNP17x581RbW6trrrlGzc3N+uSTT3TCCSfoF7/4hZ5//vkevvOOMrnYyaWS5iQ4\nXy3pzZj2eklHZ/C+SZFYABSKGQtXtyWlVo1NzZqxcHXOE9Ps2bO1fPlyrVixQnvttZd27Nih++67\nT3vssYfeffddHXPMMTrnnHM6FGkNDQ168cUXVVVVpTFjxmjJkiU6/vjjexQ7Ckc2fqkCgGSykS+H\nDBmifv36qaGhQRs3btSoUaPUr1+/Lp/38MMPa8WKFZo7d64k6YMPPtCrr76q0aNH69JLL1VTU5Pq\n6uo0cuTIHsXVlS4LOTN7VNI+CR66xt3vj1xzjaQdku5ONyAzmyhpoiQNHjy4x69DYgFQSDZsaezW\n+VT0NDFJ0mmnnaa99tpLkuTu+sEPfqDFixerV69eCoVC2rhxo/bZJz41HHXUURo0aJAkaeTIkVq7\ndi2FXAnIxi9VAJBMNvKlJE2YMEGzZ8/W22+/rUsvvTSl57i7fv7zn6u2trbDY4sXL9aCBQt0ySWX\n6Lvf/a4uvvjitOJLpMs5cu5+qrsfluCrtYi7RNLZki5yd0/wEiFJ+8W0B0XOJbvfTHevcfeaAQMG\ndOvNxOossQBAvqmqrOjW+VS1Jqb/+7//SzkxSdKuu+7adnz33Xdr06ZNeu655/T8889r77331rZt\n2zo8p3fv3m3HZWVlXc6vQ3HI1i9VAJBItvLlueeeq4ceekhLly5NWJhJ0u67766PPvqorV1bW6tf\n/epXampqkiS98sor+vjjj7Vu3Trtvffe+ta3vqUJEyZo+fLlkqTy8vK2azMh3VUrz5D0X5LOcfdP\nkly2VNKBZjbUzHaWdIGk+encNxUkFgCFZFLtMFWUl8Wdqygv06TaYWm9bk8SU3sffPCBBg4cqPLy\ncv3tb3/TunXr0ooJxSVbv1QBQCLZypc777yzvvCFL+grX/mKysrKEl4zYsQIlZWV6fDDD9ett96q\nCRMm6JBDDtERRxyhww47TJdffrl27Nihxx9/XIcffrhGjRqlOXPmtM1NnzhxokaMGJEfi51I+oWk\n3pIeicyVeNrdrzCzKkmz3P2L7r7DzK6UtFBSmaQ73f3FNO/bparKCoUSFG0kFgD5qHUIWqbn9bYm\npsrKypQS0yWXXKK+ffvGPX7RRRfpS1/6koYPH66amhoddNBBacWE4jKpdljcVAYpM79UAUAi2cqX\nLS0tevrpp/WnP/0p6TXl5eVatGhR3Lkbbrihw7YC48eP1/jx4zs8/8Ybb9SNN96YVpyxLPFoyPxQ\nU1Pjy5Yt69Fz28+Rk8KJZdq44YzZB5ATq1at0sEHHxxoDC0tLTriiCP0pz/9SQceeGCgsbRK9H0x\ns+fcvSagkApOOvkxG1hcDEA6gs6XL730ks4++2yde+65uuWWWwKLQ+pejszkqpV5JVvVOgAUitjE\nlC9FHIpT3ahq8iuAgnXIIYfo9ddfb2uvXLlS3/jGN+Ku6d27t5555plch9apoi3kJBILgNJWqIkJ\nAIAgDR8+PKP7vWVLURdyAICoQklMAACga2mtWgkA6Fw+z0MOAt8PAEAi5Ifufw8o5AAgS/r06aPN\nmzeTnCLcXZs3b1afPn2CDgUAkEfIlz3LkQytBIAsGTRokNavX69NmzYFHUre6NOnjwYNGhR0GACA\nPEK+DOtujqSQA4AsKS8v19ChQ4MOAwCAvEa+7BmGVgIAAABAgaGQAwAAAIACQyEHAAAAAAXG8nl1\nGDPbJGld0HFkSH9J7wYdRAbxfvJbsb0fqfjeE+8n3v7uPiBTwRS7PMuPxfZ3ORv4HnWN71HX+B51\nrVi/RwlzZF4XcsXEzJa5e03QcWQK7ye/Fdv7kYrvPfF+UCz42XeN71HX+B51je9R10rte8TQSgAA\nAAAoMBRyAAAAAFBgKORyZ2bQAWQY7ye/Fdv7kYrvPfF+UCz42XeN71HX+B51je9R10rqe8QcOQAA\nAAAoMPTIAQAAAECBoZDLEjM738xeNLMWM0u6eo6ZrTWzlWb2vJkty2WM3dGN93OGma02szVmNjmX\nMXaHme1lZo+Y2auRP/smua458rN53szm5zrOrnT1/Taz3mY2J/L4M2Y2JPdRpi6F93OJmW2K+ZlM\nCCLOVJnZnWb2jpm9J8OZAwAABJJJREFUkORxM7OfRd7vCjM7ItcxdkcK7+ckM/sg5uczNdcxIvuK\nLb9lQ7HlzGwoljycDcWW27Oh2H5f6CkKuex5QdI4SYtTuPYL7j4yz5dL7fL9mFmZpNslnSnpEEkX\nmtkhuQmv2yZLeszdD5T0WKSdSGPkZzPS3c/JXXhdS/H7fZmk9939AEm3Sroxt1Gmrht/f+bE/Exm\n5TTI7pst6YxOHj9T0oGRr4mSfpWDmNIxW52/H0l6Mubn8+McxITcK7b8lg3FljOzoeDzcDYUW27P\nhiL9faFHKOSyxN1XufvqoOPIlBTfz1GS1rj76+7+qaR7JY3NfnQ9MlbSXZHjuyTVBRhLT6Xy/Y59\nn3MlnWJmlsMYu6OQ/v6kxN0XS3qvk0vGSvqthz0tqdLM9s1NdN2XwvtBCSi2/JYNRZgzs6EY8nA2\nFFtuz4ZS/7fThkIueC7pYTN7zswmBh1MmqolvRnTXh85l4/2dve3IsdvS9o7yXV9zGyZmT1tZvmW\nZFL5frdd4+47JH0gqV9Oouu+VP/+nBcZhjjXzPbLTWhZU0j/ZlJ1rJn908z+amaHBh0MAlVM+S0b\nivHff3cUQx7OhmLL7dlQir8vJLRT0AEUMjN7VNI+CR66xt3vT/Fljnf3kJkNlPSImb0c+dQ75zL0\nfvJGZ+8ntuHubmbJlm/dP/Lz+YykRWa20t1fy3SsSNlfJP3B3beb2eUKfyJ5csAxIWq5wv9mtprZ\nFyXVKzxsFAWm2PJbNhRbzswG8jACVBK/L1DIpcHdT83Aa4Qif75jZvcp3F0cSKLLwPsJSYr9xGNQ\n5FwgOns/ZrbRzPZ197ciQ9neSfIarT+f183scUmjJOVLAknl+916zXoz20nSnpI25ya8buvy/bh7\nbOyzJN2Ug7iyKa/+zaTL3T+MOX7QzH5pZv3d/d0g40L3FVt+y4Ziy5nZUAJ5OBuKLbdnQyn+vpAQ\nQysDZGa7mtnurceSTld4gnShWirpQDMbamY7S7pAUr6uMDVf0vjI8XhJHT49NbO+ZtY7ctxf0hhJ\nL+Uswq6l8v2OfZ9flrTI83fzyC7fT7v5Y+dIWpXD+LJhvqSLLewYSR/EDDUqOGa2T+s8DTM7SuEc\nU0q/XCCiCPNbNhRSzsyGYsjD2VBsuT0bSvH3hcTcna8sfEk6V+Exu9slbZS0MHL+/2/nfnEqh6IA\nDv9waMYhUSzgKZYwgk1gEKxiDJvAI/CjJsFOxhGC4Y+eRWAQ7woSSEggL+W+fF9yk9vWnHPa5vY0\nTfer32N+UN2Mcdf6c4zFY/9sPmP7Z3Xf+m3Zd87nR+u/ZD1Uf6q9sX9VXYz5UXU7zs9tdbJ03O/k\n8abe1a/qeMx3q6vqsfpXHSwd8xfzOR/3yk11XR0uHfMH+VxW/6vncf+cVKfV6Ti+0/rPW0/jGlst\nHfMX8zl7dX7+VkdLx2xs5DrYqvVtqRqN7SnWzA3VaCvW4Q3VZqvW9oVqNNXzwmfHzkgWAACASfi0\nEgAAYDIaOQAAgMlo5AAAACajkQMAAJiMRg4AAGAyGjkAAIDJaOQAAAAmo5EDAACYzAuK+FI2SR8c\nfgAAAABJRU5ErkJggg==\n","text/plain":["<Figure size 1080x360 with 2 Axes>"]},"metadata":{"tags":[]}}]},{"cell_type":"markdown","metadata":{"id":"a2oKRRjlXoDA","colab_type":"text"},"source":["## Interpretability"]},{"cell_type":"markdown","metadata":{"id":"83DaPyLrrqmE","colab_type":"text"},"source":["Since we standardized our inputs and outputs, our weights were fit to those standardized values. So we need to unstandardize our weights so we can compare it to our true weight (3.5).\n","\n","Note that both X and y were standardized.\n","\n","$\\hat{y}_{scaled} = b_{scaled} + \\sum_{j=1}^{k}W_{{scaled}_j}x_{{scaled}_j}$\n","* $y_{scaled} = \\frac{\\hat{y} - \\bar{y}}{\\sigma_y}$\n","* $x_{scaled} = \\frac{x_j - \\bar{x}_j}{\\sigma_j}$\n","\n","$\\frac{\\hat{y} - \\bar{y}}{\\sigma_y} = b_{scaled} + \\sum_{j=1}^{k}W_{{scaled}_j}\\frac{x_j - \\bar{x}_j}{\\sigma_j}$\n","\n","$ \\hat{y}_{scaled} = \\frac{\\hat{y}_{unscaled} - \\bar{y}}{\\sigma_y} = {b_{scaled}} + \\sum_{j=1}^{k} {W}_{{scaled}_j} (\\frac{x_j - \\bar{x}_j}{\\sigma_j}) $\n","\n","$\\hat{y}_{unscaled} = b_{scaled}\\sigma_y + \\bar{y} - \\sum_{j=1}^{k} {W}_{{scaled}_j}(\\frac{\\sigma_y}{\\sigma_j})\\bar{x}_j + \\sum_{j=1}^{k}{W}_{{scaled}_j}(\\frac{\\sigma_y}{\\sigma_j})x_j $\n","\n","In the expression above, we can see the expression $\\hat{y}_{unscaled} = W_{unscaled}x + b_{unscaled} $ where\n","\n","* $W_{unscaled} = \\sum_{j=1}^{k}{W}_j(\\frac{\\sigma_y}{\\sigma_j}) $\n","\n","* $b_{unscaled} = b_{scaled}\\sigma_y + \\bar{y} - \\sum_{j=1}^{k} {W}_j(\\frac{\\sigma_y}{\\sigma_j})\\bar{x}_j$"]},{"cell_type":"code","metadata":{"id":"uROsFZxYrZYu","colab_type":"code","outputId":"e0d1d937-e630-42c3-e9d5-07c206332dd0","executionInfo":{"status":"ok","timestamp":1583943490087,"user_tz":420,"elapsed":3685,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Unscaled weights\n","W_unscaled = W * (y_std/X_std)\n","b_unscaled = b * y_std + y_mean - np.sum(W_unscaled*X_mean)\n","print (\"[actual] y = 3.5X + noise\")\n","print (f\"[model] y_hat = {W_unscaled[0][0]:.1f}X + {b_unscaled[0]:.1f}\") "],"execution_count":145,"outputs":[{"output_type":"stream","text":["[actual] y = 3.5X + noise\n","[model] y_hat = 3.4X + 7.8\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"6nAJqYIwrxLW","colab_type":"text"},"source":["# PyTorch"]},{"cell_type":"markdown","metadata":{"id":"FP6JKnmtrzIK","colab_type":"text"},"source":["Now that we've implemented linear regression with Numpy, let's do the same with PyTorch. "]},{"cell_type":"code","metadata":{"id":"F0B3KaLNZPAn","colab_type":"code","colab":{}},"source":["import torch"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"jgAyIZlTxFkX","colab_type":"code","outputId":"b9744978-51ec-458b-e93b-71a7908237ef","executionInfo":{"status":"ok","timestamp":1583943490088,"user_tz":420,"elapsed":3667,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Set seed for reproducibility\n","torch.manual_seed(SEED)"],"execution_count":147,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<torch._C.Generator at 0x7f9b802315f0>"]},"metadata":{"tags":[]},"execution_count":147}]},{"cell_type":"markdown","metadata":{"id":"zNJFgUY0hcDZ","colab_type":"text"},"source":["## Split data"]},{"cell_type":"markdown","metadata":{"id":"ASzn8LT2Xwb2","colab_type":"text"},"source":["When we're working with PyTorch we normally use the scikit learn's [splitting functions](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection) to split our data."]},{"cell_type":"code","metadata":{"id":"WqcmdgEChtXI","colab_type":"code","colab":{}},"source":["from sklearn.model_selection import train_test_split"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"mU9AYWbzh4JQ","colab_type":"code","colab":{}},"source":["TRAIN_SIZE = 0.7\n","VAL_SIZE = 0.15\n","TEST_SIZE = 0.15\n","SHUFFLE = True"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"xtLx53hQhCp7","colab_type":"code","colab":{}},"source":["def train_val_test_split(X, y, val_size, test_size, shuffle):\n","    \"\"\"Split data into train/val/test datasets.\n","    \"\"\"\n","    X_train, X_test, y_train, y_test = train_test_split(\n","        X, y, test_size=test_size, shuffle=shuffle)\n","    X_train, X_val, y_train, y_val = train_test_split(\n","        X_train, y_train, test_size=val_size, shuffle=shuffle)\n","    return X_train, X_val, X_test, y_train, y_val, y_test"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"VZJMdevAO8Cg","colab_type":"text"},"source":["The `train_val_test_split` function essentially splits our data twice. First, we separate out the test set. And then we separate the remaining other set into train and validation sets. "]},{"cell_type":"code","metadata":{"id":"hu0t5R7lhCnU","colab_type":"code","outputId":"b106d7ce-9641-4ee3-b5b1-e2120da41402","executionInfo":{"status":"ok","timestamp":1583943490090,"user_tz":420,"elapsed":3635,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["# Create data splits\n","X_train, X_val, X_test, y_train, y_val, y_test = train_val_test_split(\n","    X, y, val_size=VAL_SIZE, test_size=TEST_SIZE, shuffle=SHUFFLE)\n","print (f\"X_train: {X_train.shape}, y_train: {y_train.shape}\")\n","print (f\"X_val: {X_val.shape}, y_test: {y_val.shape}\")\n","print (f\"X_test: {X_test.shape}, y_test: {y_test.shape}\")"],"execution_count":151,"outputs":[{"output_type":"stream","text":["X_train: (35, 1), y_train: (35, 1)\n","X_val: (7, 1), y_test: (7, 1)\n","X_test: (8, 1), y_test: (8, 1)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"8LSpLJkgkVRn","colab_type":"text"},"source":["## Standardize data"]},{"cell_type":"markdown","metadata":{"id":"mM9C5xIeT5C8","colab_type":"text"},"source":["We can also use scikit learn to do [preprocessing and normalization](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing). "]},{"cell_type":"code","metadata":{"id":"o-8Ii-JwkVWC","colab_type":"code","colab":{}},"source":["from sklearn.preprocessing import StandardScaler"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"M9FOu9-kkVeh","colab_type":"code","colab":{}},"source":["# Standardize the data (mean=0, std=1) using training data\n","X_scaler = StandardScaler().fit(X_train)\n","y_scaler = StandardScaler().fit(y_train)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"YNWqEGsYkViI","colab_type":"code","colab":{}},"source":["# Apply scaler on training and test data\n","X_train = X_scaler.transform(X_train)\n","y_train = y_scaler.transform(y_train).ravel().reshape(-1, 1)\n","X_val = X_scaler.transform(X_val)\n","y_val = y_scaler.transform(y_val).ravel().reshape(-1, 1)\n","X_test = X_scaler.transform(X_test)\n","y_test = y_scaler.transform(y_test).ravel().reshape(-1, 1)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"aSAH9M4UnDAm","colab_type":"code","outputId":"a024f8d5-1e63-4377-b8f0-7c6c5e3222b9","executionInfo":{"status":"ok","timestamp":1583943490091,"user_tz":420,"elapsed":3598,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":119}},"source":["# Check (means should be ~0 and std should be ~1)\n","print (f\"X_train: mean: {np.mean(X_train, axis=0)[0]:.1f}, std: {np.std(X_train, axis=0)[0]:.1f}\")\n","print (f\"y_train: mean: {np.mean(y_train, axis=0)[0]:.1f}, std: {np.std(y_train, axis=0)[0]:.1f}\")\n","print (f\"X_val: mean: {np.mean(X_val, axis=0)[0]:.1f}, std: {np.std(X_val, axis=0)[0]:.1f}\")\n","print (f\"y_val: mean: {np.mean(y_val, axis=0)[0]:.1f}, std: {np.std(y_val, axis=0)[0]:.1f}\")\n","print (f\"X_test: mean: {np.mean(X_test, axis=0)[0]:.1f}, std: {np.std(X_test, axis=0)[0]:.1f}\")\n","print (f\"y_test: mean: {np.mean(y_test, axis=0)[0]:.1f}, std: {np.std(y_test, axis=0)[0]:.1f}\")"],"execution_count":155,"outputs":[{"output_type":"stream","text":["X_train: mean: -0.0, std: 1.0\n","y_train: mean: 0.0, std: 1.0\n","X_val: mean: 0.1, std: 0.6\n","y_val: mean: 0.1, std: 0.7\n","X_test: mean: -0.3, std: 0.7\n","y_test: mean: -0.3, std: 0.6\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"X3u7SbMfbqd9","colab_type":"text"},"source":["## Weights"]},{"cell_type":"markdown","metadata":{"id":"S7FufaY1bzZA","colab_type":"text"},"source":["We will be using [Linear layers](https://pytorch.org/docs/stable/nn.html#linear-layers) in our MLP implementation. These layers will act as out weights (and biases).\n","\n","$ z = XW $ "]},{"cell_type":"code","metadata":{"id":"WLyU8Y5ZrZl5","colab_type":"code","colab":{}},"source":["from torch import nn"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"7c-gJ6BgrZjk","colab_type":"code","outputId":"26660ddb-e85a-46db-aa79-b07f2a41032c","executionInfo":{"status":"ok","timestamp":1583943490092,"user_tz":420,"elapsed":3573,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":85}},"source":["# Inputs\n","N = 3 # num samples\n","x = torch.randn(N, INPUT_DIM)\n","print (x.shape)\n","print (x.numpy())"],"execution_count":157,"outputs":[{"output_type":"stream","text":["torch.Size([3, 1])\n","[[ 0.04613046]\n"," [ 0.40240282]\n"," [-1.0115291 ]]\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"fFQcVwOrhdhN","colab_type":"code","outputId":"54599f03-e3f9-47b8-b2ce-cbab17d8993e","executionInfo":{"status":"ok","timestamp":1583943490093,"user_tz":420,"elapsed":3553,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["# Weights\n","m = nn.Linear(INPUT_DIM, OUTPUT_DIM)\n","print (m)\n","print (f\"weights ({m.weight.shape}): {m.weight[0][0]}\")\n","print (f\"bias ({m.bias.shape}): {m.bias[0]}\")"],"execution_count":158,"outputs":[{"output_type":"stream","text":["Linear(in_features=1, out_features=1, bias=True)\n","weights (torch.Size([1, 1])): 0.34761226177215576\n","bias (torch.Size([1])): -0.3370760679244995\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"1dvWqTqMhdm0","colab_type":"code","outputId":"978001d8-3e67-48d1-d594-18379c738042","executionInfo":{"status":"ok","timestamp":1583943490093,"user_tz":420,"elapsed":3535,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":85}},"source":["# Forward pass\n","z = m(x) \n","print (z.shape)\n","print (z.detach().numpy())"],"execution_count":159,"outputs":[{"output_type":"stream","text":["torch.Size([3, 1])\n","[[-0.32104054]\n"," [-0.19719592]\n"," [-0.68869597]]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"t2j5_5SopsRF"},"source":["## Model"]},{"cell_type":"markdown","metadata":{"id":"phjJCXREb5hG","colab_type":"text"},"source":["Our goal is to learn a linear model $\\hat{y}$ that models $y$ given $X$. \n","\n","$\\hat{y} = XW + b$\n","* $\\hat{y}$ = predictions | $\\in \\mathbb{R}^{NX1}$ ($N$ is the number of samples)\n","* $X$ = inputs | $\\in \\mathbb{R}^{NXD}$ ($D$ is the number of features)\n","* $W$ = weights | $\\in \\mathbb{R}^{DX1}$ \n","* $b$ = bias | $\\in \\mathbb{R}^{1}$ "]},{"cell_type":"code","metadata":{"id":"Ds-jhvIhK4t6","colab_type":"code","colab":{}},"source":["from torchsummary import summary"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"cBQtSQIorZiB","colab_type":"code","colab":{}},"source":["class LinearRegression(nn.Module):\n","    def __init__(self, input_dim, output_dim):\n","        super(LinearRegression, self).__init__()\n","        self.fc1 = nn.Linear(input_dim, output_dim)\n","        \n","    def forward(self, x_in):\n","        y_pred = self.fc1(x_in)\n","        return y_pred"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"0U7wD5a6VSZ8","colab_type":"code","outputId":"4c25c8c6-e1f0-4022-d1a0-e99e6e05fca9","executionInfo":{"status":"ok","timestamp":1583943490096,"user_tz":420,"elapsed":3510,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":306}},"source":["# Initialize model\n","model = LinearRegression(input_dim=INPUT_DIM, output_dim=OUTPUT_DIM)\n","print (model.named_parameters)\n","summary(model, input_size=(INPUT_DIM,)) # -1 is the batch size"],"execution_count":162,"outputs":[{"output_type":"stream","text":["<bound method Module.named_parameters of LinearRegression(\n","  (fc1): Linear(in_features=1, out_features=1, bias=True)\n",")>\n","----------------------------------------------------------------\n","        Layer (type)               Output Shape         Param #\n","================================================================\n","            Linear-1                    [-1, 1]               2\n","================================================================\n","Total params: 2\n","Trainable params: 2\n","Non-trainable params: 0\n","----------------------------------------------------------------\n","Input size (MB): 0.00\n","Forward/backward pass size (MB): 0.00\n","Params size (MB): 0.00\n","Estimated Total Size (MB): 0.00\n","----------------------------------------------------------------\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"pYKh54SKoma8","colab_type":"text"},"source":["## Loss"]},{"cell_type":"code","metadata":{"id":"gesdrs3Vo0nq","colab_type":"code","outputId":"f5f08948-2199-4888-80e8-e5b0274ac233","executionInfo":{"status":"ok","timestamp":1583943490096,"user_tz":420,"elapsed":3480,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["loss_fn = nn.MSELoss()\n","y_pred = torch.Tensor([0., 0., 1., 1.])\n","y_true =  torch.Tensor([1., 1., 1., 0.])\n","loss = loss_fn(y_pred, y_true)\n","print('Loss: ', loss.numpy())"],"execution_count":163,"outputs":[{"output_type":"stream","text":["Loss:  0.75\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"mctyzTveoegu","colab_type":"text"},"source":["## Optimizer"]},{"cell_type":"markdown","metadata":{"id":"PZ6IZ5pYcl1a","colab_type":"text"},"source":["When we implemented linear regression with just NumPy, we used batch gradient descent to update our weights. But there are actually many different [gradient descent optimization algorithms](https://ruder.io/optimizing-gradient-descent/) to choose from and it depends on the situation. However, the [ADAM optimizer](https://ruder.io/optimizing-gradient-descent/index.html#gradientdescentoptimizationalgorithms/#adam) has become a standard algorithm for most cases. "]},{"cell_type":"code","metadata":{"id":"ViOE7L5RdRgX","colab_type":"code","colab":{}},"source":["from torch.optim import Adam"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"8EviGS0ldLvN","colab_type":"code","colab":{}},"source":["# Optimizer\n","optimizer = Adam(model.parameters(), lr=LEARNING_RATE) "],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"nXbZVkMAyIpL","colab_type":"text"},"source":["## Training"]},{"cell_type":"code","metadata":{"id":"O6zoQ-fOghLM","colab_type":"code","colab":{}},"source":["# Convert data to tensors\n","X_train = torch.Tensor(X_train)\n","y_train = torch.Tensor(y_train)\n","X_val = torch.Tensor(X_val)\n","y_val = torch.Tensor(y_val)\n","X_test = torch.Tensor(X_test)\n","y_test = torch.Tensor(y_test)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"1xwV9WvBhVaF","colab_type":"code","outputId":"bbbdfd4e-3977-490f-e806-d9e000f022c1","executionInfo":{"status":"ok","timestamp":1583943490098,"user_tz":420,"elapsed":3459,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":102}},"source":["# Training\n","for epoch in range(NUM_EPOCHS):\n","    # Forward pass\n","    y_pred = model(X_train)\n","\n","    # Loss\n","    loss = loss_fn(y_pred, y_train)\n","\n","    # Zero all gradients\n","    optimizer.zero_grad()\n","\n","    # Backward pass\n","    loss.backward()\n","\n","    # Update weights\n","    optimizer.step()\n","\n","    if epoch%20==0: \n","        print (f\"Epoch: {epoch} | loss: {loss:.2f}\")"],"execution_count":167,"outputs":[{"output_type":"stream","text":["Epoch: 0 | loss: 0.22\n","Epoch: 20 | loss: 0.03\n","Epoch: 40 | loss: 0.02\n","Epoch: 60 | loss: 0.02\n","Epoch: 80 | loss: 0.02\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"vj7cpeCsi6tf","colab_type":"text"},"source":["## Evaluation"]},{"cell_type":"code","metadata":{"id":"LeQm-H2vi6A7","colab_type":"code","colab":{}},"source":["# Predictions\n","pred_train = model(X_train)\n","pred_test = model(X_test)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"dnW9b7ZQkA2l","colab_type":"code","outputId":"c8a78cf8-c12b-4b3b-8398-d131c15c0c85","executionInfo":{"status":"ok","timestamp":1583943490099,"user_tz":420,"elapsed":3444,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Performance\n","train_error = loss_fn(pred_train, y_train)\n","test_error = loss_fn(pred_test, y_test)\n","print(f'train_error: {train_error:.2f}')\n","print(f'test_error: {test_error:.2f}')"],"execution_count":169,"outputs":[{"output_type":"stream","text":["train_error: 0.02\n","test_error: 0.01\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"HdNCjH4ZjnRu","colab_type":"text"},"source":["Since we only have one feature, it's easy to visually inspect the model."]},{"cell_type":"code","metadata":{"id":"h0a-iUi4i6Iy","colab_type":"code","outputId":"3ea87f22-ba12-42e8-836a-8628ee93f23a","executionInfo":{"status":"ok","timestamp":1583943490405,"user_tz":420,"elapsed":3718,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":336}},"source":["# Figure size\n","plt.figure(figsize=(15,5))\n","\n","# Plot train data\n","plt.subplot(1, 2, 1)\n","plt.title(\"Train\")\n","plt.scatter(X_train, y_train, label='y_train')\n","plt.plot(X_train, pred_train.detach().numpy(), color='red', linewidth=1, linestyle='-', label='model')\n","plt.legend(loc='lower right')\n","\n","# Plot test data\n","plt.subplot(1, 2, 2)\n","plt.title(\"Test\")\n","plt.scatter(X_test, y_test, label='y_test')\n","plt.plot(X_test, pred_test.detach().numpy(), color='red', linewidth=1, linestyle='-', label='model')\n","plt.legend(loc='lower right')\n","\n","# Show plots\n","plt.show()"],"execution_count":170,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAA3IAAAE/CAYAAAADjvF6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzde3hU1b3/8c+XGDEVNSKIJohQLyiK\nEozXaGvrJd4qEatVsUItRVs9bU9tjiD9We1RAbFatWqlHm/1hrUxomhjlaooRbmEAopUVFAGVESC\nIkFzWb8/9oSZITPJJHPZc3m/nofHvffsmf2dAC4+e629ljnnBAAAAADIHj38LgAAAAAA0DUEOQAA\nAADIMgQ5AAAAAMgyBDkAAAAAyDIEOQAAAADIMgQ5AAAAAMgyBDkgw5lZgZltMrMBftcCAACAzECQ\nA5IsGLrafrWaWWPY/qiufp5zrsU518s590Eq6gUAIN2S3VaGfe5cM7swmbUCmWo7vwsAco1zrlfb\ntpmtlDTWOfdCrPPNbDvnXHM6agMAIBN0ta0E0B49ckCamdl1ZjbdzB41sy8kXWhmRwfvIjaY2Voz\nu83MCoPnb2dmzswGBvcfCr7+nJl9YWb/MrNBPn4lAACSKvhYwf8zs/fM7FMze9jMioOv7Whmj5nZ\nZ8F283Uz29XMfi/pcEn3BHv2fu/vtwBSiyAH+OMsSY9I2kXSdEnNkn4hqY+kCkmnSLqkg/dfIOn/\nSeot6QNJ/5vKYgEASLNfSzpZ0rGS+ktqknRL8LWx8kaVlcprNy+X9LVz7gpJ8+T17vUK7gM5iyAH\n+ONV59zTzrlW51yjc26ec+5151yzc+49SdMkfbuD9z/hnJvvnGuS9LCkYWmpGgCA9LhU0njn3Brn\n3BZJ10r6gZmZvFDXV9I+wXZznnPuSz+LBfzAM3KAPz4M3zGzAyT9XtJhkr4h7+/m6x28/6Ow7c2S\nesU6EQCAbBIMa3tJetbMXNhLPSTtJun/JO0h6Qkz6yXpQUn/zznXkvZiAR/RIwf4w22zf7ekpZL2\ndc7tLOlqSZb2qgAA8JlzzkkKSPquc6447NcOzrlPnXNfOeeuds4dIOlbks6RdF7b2/2qG0g3ghyQ\nGXaStFHSl2Z2oDp+Pg4AgFz3J0mTzWwvSTKz3c3se8HtE81siJn1kPS5vOfMW4Pv+1jSN/0oGEg3\nghyQGa6QNFrSF/J656b7Ww4AAL66UdILkmYFZ3ieI2l48LVSSU/JazOXSnpWoXbzFkkXmdkGM7sx\nvSUD6WVe7zUAAAAAIFvQIwcAAAAAWYYgBwAAAABZhiAHAAAAAFmGIAcAAAAAWYYgBwAAAABZZju/\nC+hInz593MCBA/0uAwCQYgsWLPjUOdfX7zqyBe0jAOSPWG1kRge5gQMHav78+X6XAQBIMTNb5XcN\n2YT2EQDyR6w2kqGVAAAAAJBlCHIAAAAAkGUIcgAAAACQZQhyAAAAAJBlCHIAAAAAkGUIcgAAAACQ\nZQhyAAAAAJBlCHIAAAAAkGUyekFwAEBmqq0PaGrdcq1paFRJcZGqKwerqqzU77IAAMgb9MgBALqk\ntj6gCTVLFGholJMUaGjUhJolqq0P+F0aAAD+a26WPvww5ZchyAEAumRq3XI1NrVEHGtsatHUuuU+\nVQQAQIZ4/33pW9+Sbrgh5ZdKSpAzs3vN7BMzWxrj9ePNbKOZLQr+ujoZ1wUApN+ahsYuHQcAIC88\n+qh05JHS978v3XFHyi+XrGfk7pf0R0kPdnDObOfcGUm6HgDAJyXFRQo0NOqcxf/QrH0O1/odi7ce\nBwAg73zxhXT55dLrr0t1dVJZWVoum5QeOefcK5I+S8ZnAQAy2zUHbq+VU87Q1OduVXHjF5KkosIC\nVVcO9rkyAADS7I03vODWs6e0YEHaQpyU3mfkjjazf5vZc2Z2UBqvCwBIltGjdVLVcZKk0yc+off6\n7KXS4iJNGjmUWSsBAPmjpUWaNEn63vekKVOkadOkHXdMawnpWn5goaS9nXObzOw0SbWS9ot2opmN\nkzROkgYMGJCm8gAgP8W9jMCyZdKQId72pEnS+PGamd5Ss5KZ3SvpDEmfOOcOjvK6SbpV0mmSNksa\n45xbmN4qAQBdsnq1dNFFUmurNH++tNdevpSRlh4559znzrlNwe1nJRWaWZ8Y505zzpU758r79u2b\njvIAIC/FtYyAc9LZZ4dC3KefSuPH+1Jvlrpf0ikdvH6qvBub+8m7iXlXGmoCAHTXk09Khx0mnXCC\n9OKLvoU4KU1Bzsz2CN51lJkdEbzu+nRcGwAQXafLCDz4oNSjh1RTI91yixfqdtvNh0qzVxzPkI+Q\n9KDzzJVUbGZ7pqc6AEDcNm+WLr1U+vWvpaeekiZOlAoKfC0pKUMrzexRScdL6mNmqyX9VlKhJDnn\n/iTp+5J+ambNkholneecc8m4NgCge2IuI7Bhs+Tde/OsXSvtsUeaqso7pZLCV41dHTy21p9yAADt\nLFoknX++VF4u1ddLO+/sd0WSkhTknHPnd/L6H+UtTwAAyBBtywiEu7D+WV33/J3eTnm5NG+eD5Uh\nGp4hB4A0a22VbrtNuv56b2TKhRf6XVGEdE12AgDIMNWVgzWhZokam1pkrlXv33hm6MWPPlLtmmZN\nnTyr84lQkIiApPAHLPoHj7XjnJsmaZoklZeXM6oFAFLp44+lMWOkDRu89eG++U2/K2onncsPAAAy\nSFVZqSaNHKprX39ka4j75MjjJOdUu6a584lQkAwzJF1knqMkbXTOMawSAPz03HPeenDl5dLs2RkZ\n4iR65AAgf339taqG9w/tr1+v3Xv3ltTxRCj0ysUvjmfIn5W39MAKecsP/MifSgEA2rLFm5m5pkZ6\n9FHp29/2u6IOEeQAIB9VVEhz5oT2t5l/KuZEKDGOI7o4niF3ki5LUzkAgFjeeku64AJp3329yU2C\nNzYzGUMrASCfbA7OSNkW4j79tF2Ik7yJUKKJdRwAgKzknHT33V7v2+WXS3/9a1aEOIkgBwD5Y//9\npR139LZ3373DdeGqKwerqDByfZyiwgJVVw5OdZUAAKTH+vXSyJFekJs9Wxo7NnL5nQzH0EoAyAG1\n9QFNrVsefYbJTz+V+vYNnbxxY6dr4LS9N+ZnAgCQzWbNkkaPln7wA+mxx6SePf2uqMsIcgCQ5Wrr\nA1uXEZBCM0xKipzMpKRECsQ/62RVWSnBDQCQW5qapKuvlh58ULrvPunkk/2uqNsIcgCQJWL1ukWb\nYbJ4/UeqGn5q6MAXX0i9eqW5YgAAMsiKFd6EJrvvLtXXe//NYgQ5AMgCHfW6bTuT5MopZ4R2evSQ\nWiJDHgAAecU5rwfu17+Wfvtb6bLLsupZuFiY7AQAskBH67q1zSQ56LNARIg7/n+fI8QBAPJbQ4PX\nCzd1qvdc3OWX50SIk+iRA4Cs0NG6brf8YFjEs3BrduqjE37xF006fWi6ygMAIPPMmSONGiWdfro0\nb55UlFtL6BDkACALlBQXKRAlzJ264R1VDQ/1wu1b/ZT69e6lScwwCQDIV83N0g03SHfeKU2bJp15\nZtou3eEs0klGkAOALFBdOTjiGTkp+rNwK3yoDQCAjLFqlXThhd5yAgsXejM2p0mHs0inIMzxjBwA\nZIGqslJNGjlUpcVFOnHF65EhrrWVZ+EAAHj8cenww70euOefT2uIkzp+nj0V6JEDgCxRVVYauS7c\n3ntLK1f6Vg8AABlh0ybp5z+XXn1VevZZqbzclzI6ep49FeiRA4BscPPNkbNsOUeIAwBg3jyprMxr\nIxcu9C3ESdo6i3S8xxNFkAOATGcmXXGFt92njxfiAADIZ62t0o03ejNSXn+99H//J/Xq5WtJ1ZWD\nVVRYEHGsqLBA1ZWDU3I9hlYCQKa68kqvkWpDgAMAQAoEpIsukr7+2uuR23tvvyuSFJrQhFkrASAD\npHMa4QjhwygPO0yaPz/11wQAINM99ZR0ySXSZZdJEyZI22VWnKkqK03b8j+Z9c0BIIOkexphSd4d\nxr/8JbRPLxwAANLmzdKvfy0995xUUyMdc4zfFfmOZ+QAIIZ0TyMss1CIGzEiaSGutj6gismzNGj8\nTFVMnqXa+kBSPhcAgLRYvNhbVmDDBmnRIkJcEEEOAGJI2zTC3/1u+xkpa2uT8tFtvYqBhkY5hXoV\nCXMAgIznnHTbbdIJJ3jPjT/yiLTLLn5XlTEIcgAQQ1qmETaT/vlPb/unP036UMq09yoCAJAMn3wi\nnXGG9PDD0ty53qMH4Tc9QZADgFhSOo3wvvu274W7887EP3cb6V6cFACAhNXVScOGeb9efVXaZx+/\nK8pIBDkAiKGqrFSTRg5VaXGRTFJpcZEmjRya+EQnZtK773rbv/tdSic0SffipAAAdNtXX3nrpv7k\nJ94wyuuvlwoL/a4qYzFrJQB0IKnTCBcWSs3Nof00zEhZXTk4YuZNKbWLkwIA0C1vvy2df740aJBU\nXy/ttpvfFWU8euQAINWc83rh2kLcn/6UtmUFUtarCABAMjgn/fnP0nHHec+K/+1vhLg4JaVHzszu\nlXSGpE+ccwdHed0k3SrpNEmbJY1xzi1MxrUBIB4ZsbC35Mu6cOlcnBQAgLh99pk3jPLdd6VXXpEO\nPNDvirJKsnrk7pd0Sgevnyppv+CvcZLuStJ1AaBTvkzB39ISGeKeeILFvQEAaPPSS95kJnvvLb3+\nOiGuG5LSI+ece8XMBnZwyghJDzrnnKS5ZlZsZns659Ym4/oA0JGOpuBPSU9VBvTCAQCQkZqapGuu\nke67T7r3XumUjvqC0JF0PSNXKunDsP3VwWPtmNk4M5tvZvPXrVuXluIA5La0TcG/ZUtkiHvmGUIc\nAABt3n1XOvZYbzKT+npCXIIybrIT59w051y5c668b9++fpcDIAekbWHvorDPc046/fTkfT4AANns\nL3+RjjpKGjVKmjlT6tfP74qyXrqCXEDSXmH7/YPHACDlUrqwd0NDZC/cnDn0wgEA0GbjRunCC6VJ\nk6QXXpB+/vP2jyCgW9IV5GZIusg8R0nayPNxANIlpQt777praN856eijE/tMAAByxb/+JZWVSTvt\nJM2fLx16qN8V5ZRkLT/wqKTjJfUxs9WSfiupUJKcc3+S9Ky8pQdWyFt+4EfJuC4AxCupU/AHAlL/\n/qH9xYuloUOT89kAAGS7lhavB+7226W775aqqvyuKCcla9bK8zt53Um6LBnXAgBfMSMlAACxffCB\nN5Ryu+2khQulUtYxTZWMm+wEADLSsmWRIW7lSkIcAADhnnhCKi/3Jvv6xz8IcSmWlB45AMhp9MIB\nABDbl19Kv/iF9PLL3oyUhx/ud0V5gR45AIhl7tzIELduXdwhrrY+oIrJszRo/ExVTJ6l2nom6gUA\n5KCFC6Xhw73n4hYuJMSlET1yABBNAr1wtfUBTahZosamFklSoKFRE2qWSFLyJlwBAMBPra3SzTdL\nN94o3XabdN55fleUd+iRA4BwM2dGhrhNm7o8lHJq3fKtIa5NY1OLptYtT0aFAAD4a+1a6ZRTpCef\nlN54gxDnE4IcALQxk844I7TvnLTjjl3+mDUNjV06DgBA1njmGW8o5THHeM/EDRzod0V5iyAHIO8t\nvObmiF64p15/P6EJTUqKi7p0HACAjNfYKP3Xf0mXXy799a/SNdd4SwzAN/z0AeQ3Mw0P2x145TMq\nevptucLCbj/PVl05OOIZOUkqKixQdeXgBIsFAMAHS5dK558vDRkiLVokFRf7XRFEjxyAfHXDDRG9\ncAP/52kNvPIZSYk/z1ZVVqpJI4eqtLhIJqm0uEiTRg5lohMAQHZxTrrjDuk735GuuEJ67DFCXAah\nRw5A/tlmRsq2ABcu0efZqspKCW6QJJnZKZJulVQg6R7n3ORtXh8jaaqktjUq/uicuyetRQLAttat\nky6+WProI2nOHGm//fyuCNugRw5A/rjsssgQ55wqJr0Y9VSeZ0MymFmBpDsknSppiKTzzWxIlFOn\nO+eGBX8R4gD46x//kIYNkw46SHrtNUJchqJHDkB+iLEuHM+zIcWOkLTCOfeeJJnZY5JGSHrL16oA\nIJqvv5YmTpQefVR68EHphBP8rggdoEcOQG4788x2vXDhM1Im8jxbbX1AFZNnadD4maqYPEu19YFO\n34O8Uyrpw7D91cFj2zrbzBab2RNmtld6SgOAMP/5j3T00d5/Fy0ixGUBeuQA5K7wANezp7RlS9TT\nuvM8W219IKInL9DQqAk1S7Z+HtAFT0t61Dn3lZldIukBSd/d9iQzGydpnCQNGDAgvRUCyF3OSffd\nJ115pfS730mXXtp+FAsyEkEOQMaprQ9oat1yrWloVElxkaorB3ctHB1yiLRkSWg/gTXhYplatzxi\nOKYUmu2SIIcwAUnhPWz9FZrURJLknFsftnuPpBujfZBzbpqkaZJUXl6e/D/UAPLPhg3SJZdIb78t\nvfSS90wcsgZDKwFklLaerkBDo5xCPV1xD1s0C4W4IUNSEuKk2LNaJjrbJXLOPEn7mdkgM9te0nmS\nZoSfYGZ7hu2eKWlZGusDkK9mz/YmNNlzT+mNNwhxWYgeOQAZpds9Xb16SV9+GdpPUYBrU1JcpECU\n0MZslwjnnGs2s8sl1clbfuBe59ybZvY7SfOdczMk/dzMzpTULOkzSWN8KxhA7mtu9oZQ/vnP0j33\nSKef7ndF6CaCHICM0q2ervCx/KedJs2cmeSq2mO2S8TLOfespGe3OXZ12PYESRPSXReAPPT++9Ko\nUdJOO0n19dIee/hdERLA0EoAGSVWj1bU42btZ6RMQ4iTEpvtEgCAtHvkEenII6VzzpGee44QlwPo\nkQOQUeLu6QoPcJdeKt11V5oqDOnObJcAAKTV559Ll1/uPQdXVyeVlfldEZKEIAcgo7QFo5izVsZY\n2BsAAGzj9delCy6QTjxRWrBA2nFHvytCEhHkAGScqD1dzkk9wkaDX3edNHFiegsDACAbtLRIU6ZI\nt94q3XmndPbZfleEFCDIAch89MIBABCfDz+UfvhDr+2cP1/aa6/O34OsxGQnADJXc3NkiLvnHkIc\nAACx1NRI5eVSZaX0wguEuBxHjxyAzNSNXrja+kDsZ+sAAMhVq1ZJAwdKfftKTz/tzU6JnEePHIDM\nsnlzZIh76qm4Q9yEmiUKNDTKSQo0NGpCzRLV1gdSVysAAH77xS+8ECdJy5cT4vIIPXIAMkcCz8JN\nrVsesWSBJDU2tWhq3XJ65QAAuae5WSos3LpbMelFrZkyhxEpeSQpPXJmdoqZLTezFWY2PsrrY8xs\nnZktCv4am4zrAki92vqAKibP0qDxM1UxeVZqerg++ywyxL36apefhVvT0Nil4wAAZK3nn98a4uqv\nnqoDf/McI1LyUMI9cmZWIOkOSSdJWi1pnpnNcM69tc2p051zlyd6PQDp0zZcsa2nq61xkJS8O31J\nmpGypLhIgSihraS4qFufBwBARtp3X+ndd73tL77Q5X98Q43btH+MSMkPyeiRO0LSCufce865ryU9\nJmlEEj4XgM86Gq6YsPfeiwxxb76Z0IyU1ZWDVVRYEHGsqLBA1ZWDu/2ZAABkjI8/9trNd9/1ZqV0\nTurVixEpeSwZQa5U0odh+6uDx7Z1tpktNrMnzIy5UIEskLLGwUzaZ5/QvnPSkCEJfWRVWakmjRyq\n0uIimaTS4iJNGjmUu5EAgOz3u99Je+zhbS9YIP3971tfijXyhBEpuS9dk508LelR59xXZnaJpAck\nfTfaiWY2TtI4SRowYECaygMQTdKHKy5cKB12WGj//fdDM20lQVVZKcENAJA7WlulgoLI/W0eSaiu\nHBzxGITEiJR8kYweuYCk8B62/sFjWznn1jvnvgru3iPpMMXgnJvmnCt3zpX37ds3CeUB6K6kDlc0\niwxxziU1xAEAkFPmzAmFuJtu8trNbZ8rFyNS8lkyeuTmSdrPzAbJC3DnSbog/AQz29M5tza4e6ak\nZUm4LoAUa2sEElpk+4UXpJNOCu2vWyf16ZPkSgEAyCFHHy3Nnettr18v9e7d4emMSMlPCQc551yz\nmV0uqU5SgaR7nXNvmtnvJM13zs2Q9HMzO1NSs6TPJI1J9LoA0iOhxiFJM1ICAJAXGhqkXXf1tocP\n956HA2JIyjpyzrlnnXP7O+f2cc5dHzx2dTDEyTk3wTl3kHPuUOfcd5xzbyfjugAy1MMPR4a4L78k\nxAEA0JHbbw+FuJdfJsShU+ma7ARAvqAXDgCA+Dkn9QjrW2lpidwHYuBPCYDkuOmmyBDX1ESIAwCg\nI//+dyi0TZzYPtQBHaBHDkDi6IUDAKBrRoyQZszwtteskfbc0996kHWI/AC677//OzLEtbYS4gAA\n6MiXX3pt54wZUmmp124S4tANBDkA3WMm/eEPof0Y69sAAICgBx+UevXytp95Rlq92t96kNUYWgmg\na846S6qtDe3TAwcAQOfCb3Z+/bVUWOhfLcgJBDkA8UvSs3C19YHEFhkHACBbvPOOtP/+3vZPfyrd\neae/9SBnMLQSQOeGDYsMcc4lFOIm1CxRoKFRTlKgoVETapaotj6QnFoBAMgUP/xhKMS9+y4hDklF\njxyQQ7rb09Xh+5I8I+XUuuVqbGqJONbY1KKpdcvplQMA5IbNm6Udd/S2zbzJwIAko0cOyBHd7emK\n9b7moqKk9cKFW9PQ2KXjAABklauuCoW4yy8nxCFl6JEDckR3e7qivW/ZdaeGdnbfXfr446TVWVJc\npECU0FZSXJS0awAA4IvwG6AbN0o77+xfLch59MgBOaK7PV3hr6+ccoZWTjkj9KJzSQ1xklRdOVhF\nhQURx4oKC1RdOTip1wEAIG0WLWo/ioUQhxQjyAE5IlaPVmc9XW2vhwe4BSUHqGLSi8krLkxVWakm\njRyq0uIimaTS4iJNGjmU5+MAANmpXz+prMzb/vvfWZYHacPQSiBHVFcO1oSaJRHDJOPp6XptwgkR\n+wOvfEZFhQWalMIesqqyUoIbACC7NTVJ228f2ifAIc3okQNyRLd6usKGgTx+zFkadOUz9JABANCZ\nP/whFOLOPpsQB1/QIwfkkLh7uqIsKXCupHNTUhUAADkkvA396CNvaCXgA3rkAJ/U1gdUMXmWBo2f\nqYrJs9KzIHZra2QDNGUKdxEBAIjHu++2n9CEEAcf0SMH+KBt7ba259na1m6TlLohjUle2BsAgLxx\n+OHS/Pne9iOPSOef3+lbausDmlq3XGsaGlVSXKTqysE8toCkokcO8EFHa74lXVNTZIi7/35CHAAA\n8WgbydIW4lpa4g5xE2qWKNDQKKfQDdu0jL5B3iDIAT7o7ppvXWbWfkat0aOTew0AAHLRQw9JBcF1\nT486ymtDe8T3T+e03rBF3mJoJeCDkuIiBaKEts7WfIvbpk3STjuF9p99Vjr11OR8NgAAuS58JMt7\n70mDBnXp7Wm7YYu8Ro8c4IPqysEqKiyIOBbPmm9xMYsMcc4R4gAAiMdHH7Wf0KSLIU6KfWM2aTds\nARHkAF90a823zmzb+Lz+Os/CAQAQr6oqac89ve3bb0+oDU3pDVsgiKGVgE86WvOtyzNdMSMlAADd\ns+2zb19/LRUWJvSRbW02s1YilQhyQIbp0tIE77wj7b9/aH/JEungg9NVKgAA2e3ZZ6XTT/e2+/eX\nPvwwaR/d0Q1bIBkYWglkmLhnujKLDHHOEeKADGRmp5jZcjNbYWbjo7ze08ymB19/3cwGpr9KIA+Z\nhULc4sVJDXFAOhDkgAzT6UxXb7wROZTygw8YSglkKDMrkHSHpFMlDZF0vpkN2ea0H0va4JzbV9It\nkqakt0ogzzQ0tJ/QZOhQ/+oBuokgB2SYDme6MpOOPDJ00Dlpr73SVBmAbjhC0grn3HvOua8lPSZp\nxDbnjJD0QHD7CUknmG374CuApPjpT6Vdd/W2r76aG6HIakkJcgwbAZIn2kxXJ69aqNcmnBA6sH49\njQ+QHUolhY/XWh08FvUc51yzpI2SdktLdUA+MZP+9Cdve/Nm6dpr/a0HSFDCk52EDRs5SV4DNc/M\nZjjn3go7beuwETM7T96wkR8kem0gF20709X7U86IPIEAB+QlMxsnaZwkDRgwwOdqgCwyZ45UURHa\npx1FjkhGjxzDRoAkqyor1Wt7fhgZ4jZvpvEBsk9AUvj45/7BY1HPMbPtJO0iaf22H+Scm+acK3fO\nlfft2zdF5QI5xiwU4mbPph1FTknG8gPRho0cGesc51yzmbUNG/k0CdcHcg/rwgG5Yp6k/cxskLzA\ndp6kC7Y5Z4ak0ZL+Jen7kmY5x196ICGNjdI3vhHa568UclDGTXZiZuPMbL6ZzV+3bp3f5QDpNWlS\nZIhrbqbxAbJY8Jm3yyXVSVom6XHn3Jtm9jszOzN42v9J2s3MVkj6laR2z5oD6IJrrw2FuJ/8hHYU\nOSsZPXJdGTayuqNhI5I3dETSNEkqLy/nbx7yB71wQE5yzj0r6dltjl0dtr1F0jnprgvISeFt6YYN\nUnGxf7UAKZaMHrmtw0bMbHt5w0ZmbHNO27ARiWEjQKSf/Syy4WltJcQBANAVS5e2XxuOEIccl3CP\nXPCZt7ZhIwWS7m0bNiJpvnNuhrxhI38JDhv5TF7YA0AvHAAAiRk4UFq1ytt++mnpjDM6PB3IFckY\nWsmwEaCrxoyRHnggtE+AAwCga5qbpcLC0H5ra/sbpEAOy7jJToCcZ0aIAwAgEXfeGQpxp5/utaWE\nOOSZpPTIAYjD8cdLL78c2ifAAQDQdeGBLRCQSkr8qwXwEUEOSIfwRqdvX+mTT/yrBQCAbLRypTRo\nUGifG6LIcwytBFJp4MD2s2gR4gAA6Jpjjw2FuAceIMQBokcOSJ3wAHfkkdLcuf7VAgBANnJO6hHW\n79DSErkP5DGCHPJKbX1AU+uWa01Do0qKi1RdOVhVZaXJvQhLCgAAkLjp06XzgitWDRsm1df7Ww+Q\nYQhyyBu19QFNqFmixqYWSVKgoVETapZIUvLCXHiIO/dcrxECAABdE96evvOOtO++/tUCZCiCHPLG\n1LrlW0Ncm8amFk2tW554kIvRC5eWHkAAAHLFunXS7ruH9hnVAsTEIGPkjTUNjV06Hpdt16254oqI\nEDehZokCDY1yCvUA1tYHukBCx8AAACAASURBVH89AABy1XnnhULcTTcR4oBO0COHvFFSXKRAlNBW\nUlzUvQ/s5Fm4lPYAAgCQ4bo0KiW8Tf3qK2n77dNTJJDF6JFD3qiuHKyiwoKIY0WFBaquHNy1D2pt\njWxwbr456l3DlPQAAgCQBeIelfKPf4Ta1N69vfaUEAfEhR455I22u4AJPbPWhRkpk94DCABAlohr\nVEp4m7pwoVRWlsYKgexHkENeqSor7d6wxq+/lnr2DO0//LB0wQUdvqW6cnDELJlSN3sAAQDIMh2O\nSvniC2nnnUMHeRYO6BaGVgKdMYsMcc51GuIkLzROGjlUpcVFMkmlxUWaNHIoz8cBAHJerNEnU2bf\nGwpx//M/hDggAfTIAbFse8fw+eelk07q0kd0uwcQAIAsFm1UysopZ4RO2LRJ2nFHHyoDcgdBDoim\nC8/CAQCASOHPpfdZ9m899eCvQi/SpgJJQZADwn38sbTHHqH9+fOlww7zrx4AALJUVVmpqob3Dx34\n5z+l44/3rR4g1xDkgDb0wgEAkBybN0cOnaRNBZKOyU6Ad9+NDHHvvEODAwBAd/3gB6EQ178/bSqQ\nIvTIIb/RCwcAQPKEt6urV0ulTPgFpAo9cshPCxdGNjZr1xLiAADorrlzI9tV5whxQIrRI4f8Qy8c\nACBP1dYHNLVuudY0NKqkuEjVlYMTXyYnvF39/e+lX/0q9rkAkoYgh/zx9NPSmWeG9jdujFwnDgCA\nHFZbH4hY2y3Q0KgJNUskqXthrrlZKiwM7be2tr9ZCiBlGFqJ/GAWGeKcI8QBAPLK1LrlEQt0S1Jj\nU4um1i3v+of9+teRIc45QhyQZvTIIbf9+c/SuHGh/S1bpJ49k3qJlAxTAQAgydY0NHbpeEzhgW3Z\nMumAAxKoCkB3EeSQu9LwLFzSh6kAAJAiJcVFCkQJbSXFRfF9wNtvSwceGNrnGXPAVwytRO659trI\nENfSkrLGJqnDVAAASKHqysEqKiyIOFZUWKDqysGdv9ksFOKuuIIQB2QAeuSQW9I8I2XShqkAAJBi\nbSNFuvQ4gHNSj7D7/k1N0nb88xHIBAn9TTSz3pKmSxooaaWkc51zG6Kc1yJpSXD3A+fcmdueAyTk\nxz+W7r03tJ+mmbMSHqYCAEAaVZWVxj/0/5ZbIpcSoBcOyCiJDq0cL+lF59x+kl4M7kfT6JwbFvxF\niENymUWGuDTOnJXQMBUAADKVWSjEzZ1LiAMyUKJBboSkB4LbD0iqSvDzgPideGJkYHMu7Q1NVVmp\nJo0cqtLiIpmk0uIiTRo5lIlOAADZac2a9m3rkUf6Vw+AmBId5NzPObc2uP2RpH4xztvBzOZLapY0\n2TlXm+B1ke/S/CxcR7o0TAUAgEy1997SBx942+eeK02f7m89ADrUaZAzsxck7RHlpYnhO845Z2ax\n/jW9t3MuYGbflDTLzJY4596Ncb1xksZJ0oABAzorD/lm4EBp1arQPkM9AABIXPgN0i+/lL7xDf9q\nARCXToOcc+7EWK+Z2cdmtqdzbq2Z7SnpkxifEQj+9z0ze0lSmaSoQc45N03SNEkqLy/nX+kIyaBe\nOAAAcsIjj0ijRoX2aVuBrJHoM3IzJI0Obo+W9NS2J5jZrmbWM7jdR1KFpLcSvC7yiZnvz8IBAJBz\nzEIh7plnaFuBLJNokJss6SQze0fSicF9mVm5md0TPOdASfPN7N+S/invGTmCHDpVWx+ICHBf7P1N\nGhkAABLV0ND+Bunpp/tXD4BuSWiyE+fcekknRDk+X9LY4PYcSUMTuQ7ykFnEFKgDr3xGRYUFmlQf\nYGIRAAC668orpRtv9LaPOEJ6/XV/6wHQbYnOWgkkX9hdwpcGHaYx514rSWpsatHUuuUEOQAAuiO8\nF+7zz6WddvKvFgAJI8ghc2wzmcnAK59pd8qahsZ0VQMAQG6or5eGDw/t85gCkBMSfUYOSJxzkSHu\n0ktVMenFqKeWFBelqSgAAHLAbruFQtzzzxPigBxCjxz8FWNJger6gCbULFFjU8vWl4oKC1RdOTid\n1QFAt5lZb0nTJQ2UtFLSuc65DVHOa5G0JLj7gXPuzHTViBz29ddSz56hfQIckHPokUNS1NYHVDF5\nlgaNn6mKybO8GSc70tISGeKuuy6ikakqK9WkkUNVWlwkk1RaXKRJI4fyfByAbDJe0ovOuf0kvRjc\nj6bROTcs+IsQh8T9/vehEHfuuYQ4IEfRI4eE1W7TexZoaNSEGu/mctTgFefC3lVlpQQ3ANlshKTj\ng9sPSHpJ0pV+FYM8Ed7GfvKJ1Levf7UASCl65JCwqXXLI4ZASqEZJiNs2RLZwNx7L3cJAeSyfs65\ntcHtjyT1i3HeDmY238zmmllVjHNkZuOC581ft25d0otFlnvnnfZrwxHigJxGjxwSFmsmyYjjcfbC\nAUA2MbMXJO0R5aWJ4TvOOWdmsf7Ht7dzLmBm35Q0y8yWOOfe3fYk59w0SdMkqby8nP+JImT4cG9m\nSkmaPt0bTgkg5xHkkLCS4iIFooS5kuIiaf16qU+f0MGZM6XTTktjdQCQOs65E2O9ZmYfm9mezrm1\nZranpE9ifEYg+N/3zOwlSWWS2gU5oJ3WVqmgIHJ/2xunAHIWQyuRsOrKwSoqLIg4VlRYoNcmnBAZ\n4pwjxAHIJzMkjQ5uj5b01LYnmNmuZtYzuN1HUoWkt9JWIbLXgw+GQlxFRfulfADkPHrkkLC2CUmm\n1i3XmoZGDXcb9bfrRoVOmD1bOvZYn6oDAN9MlvS4mf1Y0ipJ50qSmZVLutQ5N1bSgZLuNrNWeTdX\nJzvnCHLoWHhge/99aeBA30oB4B+CHJJi6wyTPAsHAJIk59x6SSdEOT5f0tjg9hxJQ9NcGrLVmjVS\nadhszrSxQF5jaCWSY+nSyBD35ps0MAAAJMsZZ4RC3B130MYCoEcOSdBJL1xtfWDrsMuS4iJVVw5m\nfTgAAOLhnNQj7L57U5O0Hf98A0CPHBIxe3ZkiFu7NmqIm1CzRIGGRjmFFguvrQ+kt1YAALLNM8+E\nQtzee3ttLCEOQBD/N0D3xPksXEeLhdMrBwBADOHt7NKl0kEH+VcLgIxEkMtRKRvOWFMjnX12aH/j\nRmnnnWOeHtdi4QAAwLNhg9S7d2ifZ+EAxMDQyixWWx9QxeRZGjR+piomz9o6XDFlwxnNIkOccx2G\nOCm4KHgXjgMAkLfGjQuFuGuuIcQB6BA9clmqLay1DVtsC2tSCoYz3nWX9LOfhfa/+krafvu43lpd\nOTiiTslbLLy6cnDX6wAAIFeFD6XcvFkq4oYngI4R5LJUR2EtqcMZE1wXbtvFwpm1EgCAMK++Kh13\nnLdtJrW2+lsPgKxBkMtSHYW1kuIiBaK83qXhjL/5jXT99aH9lpbI6Y+7YOti4QAAICT8ZumcOdLR\nR/tXC4CsQ5DLUh2FtYSHMybYCwcAADqwebO0446hfdpZAN3AZCdZqrpysIoKCyKOtYW1qrJSTRo5\nVKXFRTJJpcVFmjRyaOe9Yj/8YWSIc47GBQCAZLr66lCI++lPaWcBdBs9clmqs2fPujyckV44AABS\nK7ytbWiQdtnFv1oAZD2CXBZLyrNnxx4rvfZaaJ8ABwBAci1ZIh1ySGifthZAEhDk8hm9cAAApFb/\n/lIguI7rzJnSaaf5Ww+AnEGQy0fl5dKCBaF9AhwAAMnV1BS55mpra/sbqACQgIQmOzGzc8zsTTNr\nNbPyDs47xcyWm9kKMxufyDWRILNQiOvXjxAHAECy3X57KMRVVXltLSEOQJIl2iO3VNJISXfHOsHM\nCiTdIekkSaslzTOzGc65txK8NrqiuFjauDG0T4ADACD5wgPb2rXSHnv4VwuAnJZQkHPOLZMk6/gu\n0xGSVjjn3gue+5ikEZIIcnGorQ/EnJkybuG/PyedJD3/fHKLBAAgx3S5/X3/femb3wztc8MUQIql\nYx25Ukkfhu2vDh5DJ2rrA5pQs0SBhkY5SYGGRk2oWaLa+kB8H2DWfl04QhwAAB3qcvt79NGhEPfQ\nQ4Q4AGnRaZAzsxfMbGmUXyNSUZCZjTOz+WY2f926dam4RNaYWrdcjU0tEccam1o0tW55528OD3A/\n/jGNCgAAcYq7/W2bwGTuXG+/pUUaNSpNVQLId50OrXTOnZjgNQKS9grb7x88Fut60yRNk6Ty8vK8\nTh9rGhq7dFxSXEsKJGW4JgAAOSqu9vfRR6ULLvC2y8ulefPSUBkAhKRjaOU8SfuZ2SAz217SeZJm\npOG6Wa+kuCj+49vOiDVpUswQl9BwTQAAclyn7a9ZKMStWEGIA+CLRJcfOMvMVks6WtJMM6sLHi8x\ns2clyTnXLOlySXWSlkl63Dn3ZmJl54fqysEqKiyIOFZUWKDqysGRJ5pJPcJ+K52Txkdf5SGh4ZoA\nAOSBWO3vb8p7t3/2fJ990lwdAHgSCnLOuSedc/2dcz2dc/2cc5XB42ucc6eFnfesc25/59w+zrnr\nEy06X1SVlWrSyKEqLS6SSSotLtKkkUNDwyBbWiIblHvv7fRZuG4N1wQAII9Ea39fnHObTj2pzDvh\nD3/g2XMAvkt0HTmkWFVZafTn1+J4Fi6akuIiBaKEtljDSAAAyEdb21/nIke9fP21VFjoX2EAEJSO\nZ+SQTF99FRniZs3q0l3BuIdrAgCQ7+rqQiGuXz+vvSXEAcgQ9Mhlk272woVr691j1koAADrwgx9I\njz/ubS9aJB16qL/1AMA2CHLZ4PPPpV12Ce0vXCiVlXX742IO1wQAIN81Nkrf+Ia3XVwsbdjgbz0A\nEANDKzOdWWSIcy6hEAcAAGJ47LFQiHvySUIcgIxGj1ym+vhjaY89QvvvvCPtu69/9QAAkMu2286b\nDVrynkfffnt/6wGATtAjl4nMIkOcc4Q4AABS4f33vXa3pUX60Y+8NpcQByALEOQySVtj0mbtWtap\nAQAgVX72M+mb3/S2//Mfbz1WAMgSDK3MFEmYkRIAAMShpcUbStmGNhdAFqJHzm//+U9kiNuwgQYF\nAIBUqa8PhbgHHqDNBZC1CHJ+MpMGhy3E7Zw31TEAAEi+Sy+Vhg+XJB3/v89p0Fu7qWLyLNXWB3wu\nDAC6jqGVfliwQCovD+1v2SL17OlfPQAA5LING6TevSVJb/7XeH1/l2+rcZM3Q2WgoVETapZIEmus\nAsgq9Mh1UW19QBWTZ2nQ+Jndu4tnFhninCPEAQCQKo89tjXE6f33Na7kJDU2tUSc0tjUoql1y30o\nDgC6jyDXBbX1AU2oWaJAQ6OcQnfx4gpz8+dHPgvX3My4fAAAUqW1VRoyRDr/fOnb3/b2Bw7UmobG\nqKfHOg4AmYog1wVT65Z37y6emXT44d72zjt7Aa6gIEVVAgCQ55Yt89rZZcukmTOll17aejO1pLgo\n6ltiHQeATEWQ64Iu38WbNSuyF661Vdq4MQWVAQAASdKVV3o9cZK0aZN02mkRL1dXDlZRYeTN1KLC\nAlVXDhYAZBOCXBd06S6emXTCCd72yJFeL9y2a8UBAIDk2LTJa2dvvFG66iqv3d1xx3anVZWVatLI\noSotLpJJKi0u0qSRQ5noBEDWYdbKLqiuHKwJNUsihle2u4tXWyuddVZon+fgAABIraefls4809t+\n++3IpX2iqCorJbgByHr0yHVBp3fxzEIh7tJLCXEAkMfM7Bwze9PMWs2svIPzTjGz5Wa2wszGp7PG\nrOecVFHhhbhDDvEeYegkxAFArqBHroui3sW7/37pRz8K7XcxwNXWBzS1brnWNDSqpLhI1ZWDuVMI\nANlvqaSRku6OdYKZFUi6Q9JJklZLmmdmM5xzb6WnxCz23nvSPvt4248/Lp1zjr/1AECa0SOXKLNQ\niJs4sVshrttLGgAAMpZzbplzrrPFyY6QtMI5955z7mtJj0kakfrqstz114dC3IYNhDgAeYkg1123\n3BI5eYlz0nXXdfljur2kAQAgF5RK+jBsf3XwGKJpbPTa3t/8RrrsMq/tLS72uyoA8AVDK7sjPMDd\nfLP03//d7Y9iYVIAyF5m9oKkPaK8NNE591SSrzVO0jhJGjBgQDI/Oju8+KJ04one9qJF0qGH+lsP\nAPiMINcV994r/fjHof0kTGZSUlykQJTQxsKkAJD5nHMnJvgRAUl7he33Dx6Ldq1pkqZJUnl5eX7N\npnXGGd7C3gMGeM/GFRR0/h4AyHEMrYyXWSjEPfdct56Fq5g8S4PGz1TF5Flbn4FjYVIAyGvzJO1n\nZoPMbHtJ50ma4XNNmSMQ8NrfmTO9m6mrVhHiACCIINeZm29u/yzcKad06SM6mtCEhUkBIDeZ2Vlm\ntlrS0ZJmmlld8HiJmT0rSc65ZkmXS6qTtEzS4865N/2qOaPcdpvUv7+3/cknkbNDAwAYWtmh8AA3\ne7Z07LHd+piOJjRpW86A4AYAucU596SkJ6McXyPptLD9ZyU9m8bSMltTk7TrrtKXX0qjRkkPPeR3\nRQCQkeiRi2b69Pa9cN0McRITmgAAEJd//UvafnsvxM2ZQ4gDgA4k1CNnZudIukbSgZKOcM7Nj3He\nSklfSGqR1OycK0/kuskQdRHuYSVSj7BsW18vDRuW8LWY0ATIT01NTVq9erW2bNnidykZY4cddlD/\n/v1VWFjodynINBdeKD38sNSrl/TZZxJ/RoC8QXvp6WobmejQyqWSRkq6O45zv+Oc+zTB6yVF2zNr\nbcMdAw2Nmnf1Tap65g/eCQcfLC1ZkrTrVVcOjriexIQmQD5YvXq1dtppJw0cOFAW3sufp5xzWr9+\nvVavXq1Bgwb5XQ4yxbp10u67e9u33ir9/Of+1gMg7Wgvu9dGJhTknHPLJGXdDzz8mTVzrXr/xjND\nLwYCUklJUq/X9vxbux5AnosDctqWLVvyulHalplpt91207p16/wuBZnivvukiy/2tlevlkppF4F8\nRHvZvTYyXZOdOEnPm5mTdHdwLRzftD2bNmb+DF3zolfKG/2H6AejbtT7JSXRh10mGLqY0ATIT/nc\nKEXDzwOSpJYWadAg6cMPpdNPl555xu+KAPiM9qHrP4NOJzsxsxfMbGmUXyO6cJ1jnXPDJZ0q6TIz\n+1YH1xtnZvPNbH6q7tqW7txTNX+5YmuIG/5fD+vcUTeqpLiow6UCACCfDRw4UJ9+2vEI+XjOQZ77\n97+l7bbzQtyLLxLiAOSUdLaVnfbIOedOTPQizrlA8L+fmNmTko6Q9EqMc6dJmiZJ5eXlXVt1Ox6v\nvKJXJ54kSfr5936tGUOOlxR6Zq2zpQIAAEA3XX65dMcd3nZjo7TDDv7WAwBZLOXLD5jZjma2U9u2\npJPlTZKSXi0t0qGHSt/+tlRcrBn/WqEFx5zabhFulgoAkEtWrlypAw44QGPGjNH++++vUaNG6YUX\nXlBFRYX2228/vfHGG/rss89UVVWlQw45REcddZQWL14sSVq/fr1OPvlkHXTQQRo7dqycC91be+ih\nh3TEEUdo2LBhuuSSS9TS0hKrBEBqaPCW9bnjDum667xlfQhxADJEtraVCQU5MzvLzFZLOlrSTDOr\nCx4vMbO2xU37SXrVzP4t6Q1JM51zf0/kul32wgveMI7Fi6W//U3asEFnHrWPXhv/Xb0/+XS9Nv67\nW3vbYi0JwFIBALLVihUrdMUVV+jtt9/W22+/rUceeUSvvvqqbrrpJt1www367W9/q7KyMi1evFg3\n3HCDLrroIknStddeq2OPPVZvvvmmzjrrLH3wwQeSpGXLlmn69Ol67bXXtGjRIhUUFOjhhx/28ysi\nk/31r94C35L07rvSxIn+1gMAUWRjW5norJVPSnoyyvE1kk4Lbr8n6dBErpOQiy/2ZsUqKZHef99b\naLQDLBUAIGVS8SC363wE+qBBgzR06FBJ0kEHHaQTTjhBZqahQ4dq5cqVWrVqlf72t79Jkr773e9q\n/fr1+vzzz/XKK6+opqZGknT66adr1+A/xl988UUtWLBAhx9+uCSpsbFRu7dNHw+0cc4bCbNkiVRR\nIc2enZq/AwByjw/tZTa2lematdI/J54onX22NytWHFgqAEDKxBG6UqFnz55bt3v06LF1v0ePHmpu\nbu7y4tzOOY0ePVqTJk1Kap3IIcuXSwcc4G3PmCF973v+1gMgu/jQXmZjW5nyZ+R8d8EFcYe4NlVl\npVGHXQJALjruuOO2Dvd46aWX1KdPH+2888761re+pUceeUSS9Nxzz2nDhg2SpBNOOEFPPPGEPvnk\nE0nSZ599plWrVvlTPDLPVVeFQtwXXxDiAOSETGwrc79HDgDQoWuuuUYXX3yxDjnkEH3jG9/QAw88\nIEn67W9/q/PPP18HHXSQjjnmGA0YMECSNGTIEF133XU6+eST1draqsLCQt1xxx3ae++9/fwa8NuX\nX0q9ennb//M/0pQp/tYDAEmUiW2lOZ+G+sSjvLzczZ8/3+8yAKBbli1bpgMPPNDvMjJOtJ+LmS1w\nzpX7VFLWSUb7WFsfSN5jBM89J512mrf95pvSkCEJ1QYgv9BehnSljaRHDgCAPFNbH4iY2CvQ0KgJ\nNUskqWthzjnpO9+RXn5ZOvBAaelSqUfuP7UBAJmA/9sCAJBnptYtj5idWZIam1o0tW55/B+ycqUX\n2l5+WXr0UemttwhxAJBG/B8XAIA8s6ahsUvH25kyRRo0yNv+7DPpvPOSVBkAIF4MrQQAIM+UFBcp\nECW0lRQXdfzGr76SdtjB277kEulPf0pBdQCAeNAjBwBAnqmuHKyiwoKIY0WFBaquHBz7TS+/HApx\nCxYQ4gDAZ/TIAQCQZ9omNIl71sqqKumpp6Q995Q+/FAqKIh+HgAgbQhyAADkoaqy0s5nqFy7Viop\n8bb//Gdp7NjUFwYAGWrlypWaM2eOLrjggm69/4YbbtBVV12VtHoYWgkAeW7lypV65JFHuvXeY445\nJsnVIGPcdVcoxH30ESEOQN5LpL2UvCCXTAQ5AMgQtfUBVUyepUHjZ6pi8izV1gfSct2OGqbm5uYO\n3ztnzpxUlAQ/NTdLvXtLP/uZdO653lpx/fr5XRUAbJXs9vLqq6/WH/7wh637EydO1K233truvPHj\nx2v27NkaNmyYbrnlFrW0tKi6ulqHH364DjnkEN19992SpLVr1+pb3/qWhg0bpoMPPlizZ8/W+PHj\n1djYqGHDhmnUqFEJ1duGoZUAkAGStkBzmKuvvlq9e/fWL3/5S0lew7T77rvrF7/4RcR548eP17Jl\nyzRs2DCNHj1au+66q2pqarRp0ya1tLRo5syZGjFihDZs2KCmpiZdd911GjFihCSpV69e2rRpk156\n6SVdc8016tOnj5YuXarDDjtMDz30kMysuz8S+OGNN6Qjj/S2X31Vqqjwtx4A2EYq2suLL75YI0eO\n1C9/+Uu1trbqscce0xtvvNHuvMmTJ+umm27SM888I0maNm2adtllF82bN09fffWVKioqdPLJJ6um\npkaVlZWaOHGiWlpatHnzZh133HH64x//qEWLFnXzm7eX00Gutj4Q/4PcAOCjjhZoTnfDdP/992vh\nwoVavHixevfurebmZj355JPaeeed9emnn+qoo47SmWee2S6k1dfX680331RJSYkqKir02muv6dhj\nj+1W7fDBmDHSAw9IPXtKn38ubb+93xUBQDupaC8HDhyo3XbbTfX19fr4449VVlam3XbbrdP3Pf/8\n81q8eLGeeOIJSdLGjRv1zjvv6PDDD9fFF1+spqYmVVVVadiwYd2qqzM5G+RSkdYBIFUSXqA5iu42\nTJJ00kknqXfv3pIk55yuuuoqvfLKK+rRo4cCgYA+/vhj7bHHHhHvOeKII9S/f39J0rBhw7Ry5UqC\nXDZYv17q08fb/v3vpV/9yt96AKADqWgvJWns2LG6//779dFHH+niiy+O6z3OOd1+++2qrKxs99or\nr7yimTNnasyYMfrVr36liy66KKH6osnZZ+Q6SusAkGliLcTc6QLNnWhrmO677764GyZJ2nHHHbdu\nP/zww1q3bp0WLFigRYsWqV+/ftqyZUu79/Ts2XPrdkFBQafP1yEDPPhgKMR98AEhDkDGS1V7edZZ\nZ+nvf/+75s2bFzWYSdJOO+2kL774Yut+ZWWl7rrrLjU1NUmS/vOf/+jLL7/UqlWr1K9fP/3kJz/R\n2LFjtXDhQklSYWHh1nOTIWeDXKrSOgCkQrcWaI5DdxqmbW3cuFG77767CgsL9c9//lOrVq1KqCZk\nkNGjpcpKb0KTvfbyuxoA6FSq2svtt99e3/nOd3TuueeqIMZamYcccogKCgp06KGH6pZbbtHYsWM1\nZMgQDR8+XAcffLAuueQSNTc366WXXtKhhx6qsrIyTZ8+feuz6ePGjdMhhxzCZCedKSkuUiBKaEs0\nrQNAKnR5geY4tTVMxcXFcTVMY8aM0a677hrx+qhRo/S9731PQ4cOVXl5uQ444ICEakIGcc7vCgCg\nS1LVXra2tmru3Ln661//GvOcwsJCzZo1K+LYDTfc0G5ZgdGjR2v06NHt3j9lyhRNmTIloTrD5WyQ\nq64cHPGMnJSctA4AqRLXAs1d1N2GacyYMVu3+/Tpo3/9619R37tp0yZJ0vHHH6/jjz9+6/E//vGP\n3S8aAIAOJLu9fOutt3TGGWforLPO0n777Ze0z021nA1yqUrrAJAtsrVhAgAgnYYMGaL33ntv6/6S\nJUv0wx/+MOKcnj176vXXX093aR3K2SAnpebuNgBki2xtmAAA8NPQoUOTut5bquR0kAMAhGRLwwQA\nADqXs7NWAkAmcEwmEYGfBwAgGtqHrv8MCHIAkCI77LCD1q9fT+MU5JzT+vXrtcMOO/hdCgAgg9Be\ndq+NZGglAKRI//79tXr1aq1bt87vUjLGDjvsoP79+/tdBgAgg9BeerraRhLkACBFCgsLNWjQIL/L\nAAAgo9Fedg9DKwEAAAAgyxDkAAAAACDLEOQAAAAAIMtYJs8OY2brJK1K4yX7SPo0jddLp1z9brn6\nvSS+W7biu3XP3s65O9SBmAAABZ9JREFUvin67JzjQ/uYqXL571si+LnExs8mNn42sfn9s4naRmZ0\nkEs3M5vvnCv3u45UyNXvlqvfS+K7ZSu+G5A+/JmMjp9LbPxsYuNnE1um/mwYWgkAAAAAWYYgBwAA\nAABZhiAXaZrfBaRQrn63XP1eEt8tW/HdgPThz2R0/Fxi42cTGz+b2DLyZ8MzcgAAAACQZeiRAwAA\nAIAsk9dBzszOMbM3zazVzGLORGNmK81siZktMrP56ayxO7rwvU4xs+VmtsLMxqezxu4ys95m9g8z\neyf4311jnNcS/P1aZGYz0l1nV3T2+2BmPc1sevD1181sYPqr7J44vtsYM1sX9ns11o86u8rM7jWz\nT8xsaYzXzcxuC37vxWY2PN01dlcc3+14M9sY9nt2dbprRP7K5fYtEbnYNiYql9vWROVq25yobGzb\n8zrISVoqaaSkV+I49zvOuWGZOPVoFJ1+LzMrkHSHpFMlDZF0vpkNSU95CRkv6UXn3H6SXgzuR9MY\n/P0a5pw7M33ldU2cvw8/lrTBObevpFskTUlvld3ThT9j08N+r+5Ja5Hdd7+kUzp4/VRJ+wV/jZN0\nVxpqSpb71fF3k6TZYb9nv0tDTUCbXG7fEpFTbWOicrltTVSOt82Jul9Z1rbndZBzzi1zzi33u45k\ni/N7HSFphXPuPefc15IekzQi9dUlbISkB4LbD0iq8rGWZIjn9yH8Oz8h6QQzszTW2F3Z+mesU865\nVyR91sEpIyQ96DxzJRWb2Z7pqS4xcXw3wDc53r4lItfaxkTlctuaqHz8+xGXbGzb8zrIdYGT9LyZ\nLTCzcX4XkySlkj4M218dPJbp+jnn1ga3P5LUL8Z5O5jZfDOba2aZ3KDF8/uw9RznXLOkjfr/7d07\naxRRGMbx/wOigoioQVQUMSAIVoKIt0rEwkIQLay0sNDCb2BnI/gBrNJbKHgDRQxqKQpCCF7wVhkk\nooViIxavxZzAYm4rZzKzZ/b5wZCzFybvO2fPvjmzZyewvpHo8vT7GjuZlijclLS1mdCWXKnjq1/7\nJU1IeiBpV9vBmP2j6+NvLl2rjbm6XFtzDXNtzjVw7y3L2vzlTZA0Dmyc46FLEXGnz90ciogpSRuA\nR5Lepll7a2rKayAtlFvvjYgISfNddnVb6rNR4LGkyYj4WHeslu0ecD0ifks6T3V29HDLMdnCXlKN\nr1+SjgG3qZaZmNWiy/Uth2ujNci1uRCdn8hFxJEa9jGVfn6VdIvqY+lWJ3I15DUF9J5h2ZLua91C\nuUmalrQpIr6kj7O/zrOPmT77JOkpsBsYxGLVTz/MPOezpGXAGuB7M+FlWTS3iOjNYwy42kBcTRjY\n8ZUrIn72tO9LuiZpJCK+tRmXdUeX61uOIauNubpcW3MNc23ONXDvLV5auQhJqyStnmkDR6m+bF26\nF8AOSdslLQdOAyVcweoucDa1zwKzzs5KWitpRWqPAAeB141F+H/66YfenE8Bj6OMfwC5aG7/rC0/\nDrxpML6ldBc4k65wtQ/40bPsqWiSNs58j0TSXqo6Mgx//Fg5Sq1vObpWG3N1ubbmGubanGvwantE\nDO0GnKBa3/obmAYepvs3A/dTexSYSNsrqqUdrceem1e6fQx4R3U2buDzSjGvp7oi13tgHFiX7t8D\njKX2AWAy9dkkcK7tuBfJaVY/AJeB46m9ErgBfACeA6Ntx1xjblfSuJoAngA72465z7yuA1+AP2ms\nnQMuABfS46K6KtjH9Brc03bMNeZ2safPngEH2o7Z2/BsXa5vmcelc7WxhmPS2drawLEpsjbXcFyK\nq+1KgZmZmZmZmVkhvLTSzMzMzMysMJ7ImZmZmZmZFcYTOTMzMzMzs8J4ImdmZmZmZlYYT+TMzMzM\nzMwK44mcmZmZmZlZYTyRMzMzMzMzK4wncmZmZmZmZoX5C/kHQF5KcakkAAAAAElFTkSuQmCC\n","text/plain":["<Figure size 1080x360 with 2 Axes>"]},"metadata":{"tags":[]}}]},{"cell_type":"markdown","metadata":{"id":"e9ppP7yDmAq7","colab_type":"text"},"source":["## Inference"]},{"cell_type":"markdown","metadata":{"id":"x1EeMpH8mD3f","colab_type":"text"},"source":["After training a model, we can use it to predict on new data."]},{"cell_type":"code","metadata":{"id":"gyGJ3ossi6Gm","colab_type":"code","colab":{}},"source":["# Feed in your own inputs\n","sample_indices = [10, 15, 25]\n","X_infer = np.array(sample_indices, dtype=np.float32)\n","X_infer = torch.Tensor(X_scaler.transform(X_infer.reshape(-1, 1)))"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"9PuOcgwlmMST","colab_type":"text"},"source":["Recall that we need to unstandardize our predictions.\n","\n","$ \\hat{y}_{scaled} = \\frac{\\hat{y} - \\mu_{\\hat{y}}}{\\sigma_{\\hat{y}}} $\n","\n","$ \\hat{y} = \\hat{y}_{scaled} * \\sigma_{\\hat{y}} + \\mu_{\\hat{y}} $"]},{"cell_type":"code","metadata":{"id":"j0uQOCzai5-G","colab_type":"code","outputId":"cd831871-e656-47e0-d7fe-518a0d4fd8b3","executionInfo":{"status":"ok","timestamp":1583943490408,"user_tz":420,"elapsed":3683,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["# Unstandardize predictions\n","pred_infer = model(X_infer).detach().numpy() * np.sqrt(y_scaler.var_) + y_scaler.mean_\n","for i, index in enumerate(sample_indices):\n","    print (f\"{df.iloc[index]['y']:.2f} (actual) → {pred_infer[i][0]:.2f} (predicted)\")"],"execution_count":172,"outputs":[{"output_type":"stream","text":["35.73 (actual) → 42.11 (predicted)\n","59.34 (actual) → 59.17 (predicted)\n","97.04 (actual) → 93.30 (predicted)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"1TMS8GvcmWZ0","colab_type":"text"},"source":["## Interpretability"]},{"cell_type":"markdown","metadata":{"id":"jd_eIgQBmYzm","colab_type":"text"},"source":["Linear regression offers the great advantage of being highly interpretable. Each feature has a coefficient which signifies its importance/impact on the output variable y. We can interpret our coefficient as follows: by increasing X by 1 unit, we increase y by $W$ (~3.65) units. "]},{"cell_type":"code","metadata":{"id":"YvIKQ6sVmV-A","colab_type":"code","outputId":"395b00f7-c394-49da-f919-f236e82bffce","executionInfo":{"status":"ok","timestamp":1583943490408,"user_tz":420,"elapsed":3652,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Unstandardize coefficients \n","W = model.fc1.weight.data.numpy()[0][0]\n","b = model.fc1.bias.data.numpy()[0]\n","W_unscaled = W * (y_scaler.scale_/X_scaler.scale_)\n","b_unscaled = b * y_scaler.scale_ + y_scaler.mean_ - np.sum(W_unscaled*X_scaler.mean_)\n","print (\"[actual] y = 3.5X + noise\")\n","print (f\"[model] y_hat = {W_unscaled[0]:.1f}X + {b_unscaled[0]:.1f}\") "],"execution_count":173,"outputs":[{"output_type":"stream","text":["[actual] y = 3.5X + noise\n","[model] y_hat = 3.4X + 8.0\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"7-BrAEsOsj3g","colab_type":"text"},"source":["# Regularization"]},{"cell_type":"markdown","metadata":{"id":"rCEbjZARsmBc","colab_type":"text"},"source":["Regularization helps decrease overfitting. Below is L2 regularization (ridge regression). There are many forms of regularization but they all work to reduce overfitting in our models. With L2 regularization, we are penalizing the weights with large magnitudes by decaying them. Having certain weights with high magnitudes will lead to preferential bias with the inputs and we want the model to work with all the inputs and not just a select few. There are also other types of regularization like L1 (lasso regression) which is useful for creating sparse models where some feature cofficients are zeroed out, or elastic which combines L1 and L2 penalties. \n","\n","**Note**: Regularization is not just for linear regression. You can use it to regularize any model's weights including the ones we will look at in future lessons."]},{"cell_type":"markdown","metadata":{"id":"QfVSaAQfspyk","colab_type":"text"},"source":["$ J(\\theta) = = \\frac{1}{2}\\sum_{i}(X_iW - y_i)^2 + \\frac{\\lambda}{2}W^TW$\n","\n","$ \\frac{\\partial{J}}{\\partial{W}}  = X (\\hat{y} - y) + \\lambda W $\n","\n","$W = W- \\alpha\\frac{\\partial{J}}{\\partial{W}}$\n","* $\\lambda$ is the regularzation coefficient"]},{"cell_type":"markdown","metadata":{"id":"o2S08Iexota7","colab_type":"text"},"source":["In PyTorch, we can add L2 regularization by adjusting our optimizer. The [Adam](https://pytorch.org/docs/stable/optim.html#torch.optim.Adam) optimizer has a `weight_decay` paramter which to control the L2 penalty."]},{"cell_type":"code","metadata":{"id":"E-KcdfC1slrn","colab_type":"code","colab":{}},"source":["L2_LAMBDA = 1e-2"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"cevnJ1JZqZed","colab_type":"code","colab":{}},"source":["# Initialize model\n","model = LinearRegression(input_dim=INPUT_DIM, output_dim=OUTPUT_DIM)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"QjvyB0_vaGHw","colab_type":"code","colab":{}},"source":["# Optimizer (w/ L2 regularization)\n","optimizer = Adam(model.parameters(), lr=LEARNING_RATE, weight_decay=L2_LAMBDA) "],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"EP9k4W8spOPe","colab_type":"code","outputId":"b23d9440-495f-4646-c9ef-0b07159777c6","executionInfo":{"status":"ok","timestamp":1583943490514,"user_tz":420,"elapsed":3699,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":102}},"source":["# Training\n","for epoch in range(NUM_EPOCHS):\n","    # Forward pass\n","    y_pred = model(X_train)\n","\n","    # Loss\n","    loss = loss_fn(y_pred, y_train)\n","\n","    # Zero all gradients\n","    optimizer.zero_grad()\n","\n","    # Backward pass\n","    loss.backward()\n","\n","    # Update weights\n","    optimizer.step()\n","\n","    if epoch%20==0: \n","        print (f\"Epoch: {epoch} | loss: {loss:.2f}\")"],"execution_count":177,"outputs":[{"output_type":"stream","text":["Epoch: 0 | loss: 2.20\n","Epoch: 20 | loss: 0.06\n","Epoch: 40 | loss: 0.03\n","Epoch: 60 | loss: 0.02\n","Epoch: 80 | loss: 0.02\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"khfs3lDztn8W","colab_type":"code","colab":{}},"source":["# Predictions\n","pred_train = model(X_train)\n","pred_test = model(X_test)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"Tr2MPdIOtoIT","colab_type":"code","outputId":"88ddf4e3-4ddc-4f88-9041-ee456d55fd8a","executionInfo":{"status":"ok","timestamp":1583943490517,"user_tz":420,"elapsed":3658,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Performance\n","train_error = loss_fn(pred_train, y_train)\n","test_error = loss_fn(pred_test, y_test)\n","print(f'train_error: {train_error:.2f}')\n","print(f'test_error: {test_error:.2f}')"],"execution_count":179,"outputs":[{"output_type":"stream","text":["train_error: 0.02\n","test_error: 0.01\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"zbT59j1otfV4","colab_type":"text"},"source":["Regularization didn't make a difference in performance with this specific example because our data is generated from a perfect linear equation but for large realistic data, regularization can help our model generalize well."]},{"cell_type":"markdown","metadata":{"id":"XwSZgi_pCb3Q","colab_type":"text"},"source":["---\n","Share and discover ML projects at <a href=\"https://madewithml.com/\">Made With ML</a>.\n","\n","<div align=\"left\">\n","<a class=\"ai-header-badge\" target=\"_blank\" href=\"https://github.com/madewithml/lessons\"><img src=\"https://img.shields.io/github/stars/madewithml/lessons.svg?style=social&label=Star\"></a>&nbsp;\n","<a class=\"ai-header-badge\" target=\"_blank\" href=\"https://www.linkedin.com/company/madewithml\"><img src=\"https://img.shields.io/badge/style--5eba00.svg?label=LinkedIn&logo=linkedin&style=social\"></a>&nbsp;\n","<a class=\"ai-header-badge\" target=\"_blank\" href=\"https://twitter.com/madewithml\"><img src=\"https://img.shields.io/twitter/follow/madewithml.svg?label=Follow&style=social\"></a>\n","</div>\n","             "]}]}